[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:53:10 dist-test-slave-pbjp krb5kdc[24230](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:53:10 dist-test-slave-pbjp krb5kdc[24230](info): set up 2 sockets
Sep 28 08:53:10 dist-test-slave-pbjp krb5kdc[24230](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 08:53:10.948882 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:53:11 dist-test-slave-pbjp krb5kdc[24230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:53:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:53:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.167.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
I20260928 08:53:11.064594 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:33067
--webserver_interface=127.23.167.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:35419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:33067
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:11.145489 24243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:11.145736 24243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:11.145771 24243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:11.147056 24243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:53:11.147107 24243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:11.147126 24243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:11.147143 24243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:53:11.147157 24243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:53:11.149022 24243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:35419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:33067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:33067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:11.149375 24243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:11.149590 24243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:11.157035 24249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:11.157014 24248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:11.157159 24251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:11.157897 24243 server_base.cc:1034] running on GCE node
I20260928 08:53:11.158146 24243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:11.158496 24243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:11.159659 24243 hybrid_clock.cc:648] HybridClock initialized: now 1790585591159636 us; error 46 us; skew 500 ppm
Sep 28 08:53:11 dist-test-slave-pbjp krb5kdc[24230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585591, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:11.162333 24243 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:53:11.162777 24243 webserver.cc:460] Webserver started at http://127.23.167.254:44811/ using document root <none> and password file <none>
I20260928 08:53:11.162969 24243 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:11.163007 24243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:11.163101 24243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:11.164299 24243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "cda9fc9c7aaa407dbf9384a911ebded7"
format_stamp: "Formatted at 2026-09-28 08:53:11 on dist-test-slave-pbjp"
server_key: "8b2e70ab565ef6da2b16780f7bc7fdf3"
server_key_iv: "9bbea5fa0441e58e42aa1b9209a08029"
server_key_version: "encryptionkey@0"
I20260928 08:53:11.164888 24243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "cda9fc9c7aaa407dbf9384a911ebded7"
format_stamp: "Formatted at 2026-09-28 08:53:11 on dist-test-slave-pbjp"
server_key: "8b2e70ab565ef6da2b16780f7bc7fdf3"
server_key_iv: "9bbea5fa0441e58e42aa1b9209a08029"
server_key_version: "encryptionkey@0"
I20260928 08:53:11.166361 24243 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:53:11.167127 24258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:11.167295 24243 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:53:11.167382 24243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "cda9fc9c7aaa407dbf9384a911ebded7"
format_stamp: "Formatted at 2026-09-28 08:53:11 on dist-test-slave-pbjp"
server_key: "8b2e70ab565ef6da2b16780f7bc7fdf3"
server_key_iv: "9bbea5fa0441e58e42aa1b9209a08029"
server_key_version: "encryptionkey@0"
I20260928 08:53:11.167447 24243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:11.183063 24243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:11.185437 24243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:11.185547 24243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:11.189122 24243 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:33067
I20260928 08:53:11.189163 24310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:33067 every 8 connection(s)
I20260928 08:53:11.189487 24243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:53:11.190351 24311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:11.192744 24311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda9fc9c7aaa407dbf9384a911ebded7: Bootstrap starting.
I20260928 08:53:11.193349 24311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cda9fc9c7aaa407dbf9384a911ebded7: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:11.193599 24311 log.cc:824] T 00000000000000000000000000000000 P cda9fc9c7aaa407dbf9384a911ebded7: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:11.194289 24311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda9fc9c7aaa407dbf9384a911ebded7: No bootstrap required, opened a new log
I20260928 08:53:11.195919 24311 raft_consensus.cc:348] T 00000000000000000000000000000000 P cda9fc9c7aaa407dbf9384a911ebded7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda9fc9c7aaa407dbf9384a911ebded7" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 33067 } }
I20260928 08:53:11.196100 24311 raft_consensus.cc:374] T 00000000000000000000000000000000 P cda9fc9c7aaa407dbf9384a911ebded7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:11.196139 24311 raft_consensus.cc:729] T 00000000000000000000000000000000 P cda9fc9c7aaa407dbf9384a911ebded7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda9fc9c7aaa407dbf9384a911ebded7, State: Initialized, Role: FOLLOWER
I20260928 08:53:11.196272 24311 consensus_queue.cc:260] T 00000000000000000000000000000000 P cda9fc9c7aaa407dbf9384a911ebded7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda9fc9c7aaa407dbf9384a911ebded7" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 33067 } }
I20260928 08:53:11.196338 24311 raft_consensus.cc:388] T 00000000000000000000000000000000 P cda9fc9c7aaa407dbf9384a911ebded7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:53:11.196367 24311 raft_consensus.cc:482] T 00000000000000000000000000000000 P cda9fc9c7aaa407dbf9384a911ebded7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:53:11.196413 24311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cda9fc9c7aaa407dbf9384a911ebded7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:11.196918 24311 raft_consensus.cc:504] T 00000000000000000000000000000000 P cda9fc9c7aaa407dbf9384a911ebded7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda9fc9c7aaa407dbf9384a911ebded7" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 33067 } }
I20260928 08:53:11.197009 24311 leader_election.cc:302] T 00000000000000000000000000000000 P cda9fc9c7aaa407dbf9384a911ebded7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cda9fc9c7aaa407dbf9384a911ebded7; no voters:
I20260928 08:53:11.197191 24311 leader_election.cc:288] T 00000000000000000000000000000000 P cda9fc9c7aaa407dbf9384a911ebded7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:53:11.197232 24314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cda9fc9c7aaa407dbf9384a911ebded7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:53:11.197381 24314 raft_consensus.cc:686] T 00000000000000000000000000000000 P cda9fc9c7aaa407dbf9384a911ebded7 [term 1 LEADER]: Becoming Leader. State: Replica: cda9fc9c7aaa407dbf9384a911ebded7, State: Running, Role: LEADER
I20260928 08:53:11.197446 24311 sys_catalog.cc:564] T 00000000000000000000000000000000 P cda9fc9c7aaa407dbf9384a911ebded7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:53:11.197472 24314 consensus_queue.cc:237] T 00000000000000000000000000000000 P cda9fc9c7aaa407dbf9384a911ebded7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda9fc9c7aaa407dbf9384a911ebded7" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 33067 } }
I20260928 08:53:11.197809 24316 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda9fc9c7aaa407dbf9384a911ebded7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cda9fc9c7aaa407dbf9384a911ebded7. Latest consensus state: current_term: 1 leader_uuid: "cda9fc9c7aaa407dbf9384a911ebded7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda9fc9c7aaa407dbf9384a911ebded7" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 33067 } } }
I20260928 08:53:11.197883 24316 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda9fc9c7aaa407dbf9384a911ebded7 [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:11.197939 24315 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda9fc9c7aaa407dbf9384a911ebded7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cda9fc9c7aaa407dbf9384a911ebded7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda9fc9c7aaa407dbf9384a911ebded7" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 33067 } } }
I20260928 08:53:11.197989 24315 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda9fc9c7aaa407dbf9384a911ebded7 [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:11.198120 24320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:53:11.198642 24320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:53:11.199738 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 24243
I20260928 08:53:11.199846 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:53:11.200502 24223 external_mini_cluster.cc:1428] Setting key a1045a817c74dcf0013c522551edd7d9
I20260928 08:53:11.200990 24320 catalog_manager.cc:1347] Generated new cluster ID: c675e733ffdb4fe9b96979d17cff66d0
I20260928 08:53:11.201031 24320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:53:11.210696 24320 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:53:11.211215 24320 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 08:53:11 dist-test-slave-pbjp krb5kdc[24230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:11.214761 24320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cda9fc9c7aaa407dbf9384a911ebded7: Generated new TSK 0
I20260928 08:53:11.214939 24320 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:53:11.219035 24333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:11.201309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48774 (local address 127.23.167.254:33067)
0928 08:53:11.201456 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:11.201465 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:11.201798 (+ 333us) server_negotiation.cc:407] Connection header received
0928 08:53:11.204050 (+ 2252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:11.204058 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:11.204119 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:11.204273 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:11.206318 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:11.208016 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:11.209571 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:11.210348 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:11.210462 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:11.215204 (+ 4742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:11.215209 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:11.215209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:11.215223 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:11.217032 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:11.217505 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:11.217509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:11.217509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:11.217626 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:11.217842 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:11.217843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:11.217843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:11.218163 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:11.218311 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:11.218623 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 08:53:11.218790 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for kudu/127.23.167.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
I20260928 08:53:11.244444 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:0
--local_ip_for_outbound_sockets=127.23.167.193
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:33067
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:35419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:11.328550 24337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:11.328768 24337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:11.328804 24337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:11.330027 24337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:11.330070 24337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:11.330097 24337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:53:11.331523 24337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:35419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:33067
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:11.331722 24337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:11.331911 24337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:11.336944 24342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:11.336948 24343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:11.336982 24337 server_base.cc:1034] running on GCE node
W20260928 08:53:11.336948 24345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:11.337388 24337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:11.337638 24337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:11.338825 24337 hybrid_clock.cc:648] HybridClock initialized: now 1790585591338810 us; error 25 us; skew 500 ppm
Sep 28 08:53:11 dist-test-slave-pbjp krb5kdc[24230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585591, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:11.341387 24337 init.cc:377] Logged in from keytab as kudu/127.23.167.193@KRBTEST.COM (short username kudu)
I20260928 08:53:11.341812 24337 webserver.cc:460] Webserver started at http://127.23.167.193:39163/ using document root <none> and password file <none>
I20260928 08:53:11.342047 24337 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:11.342097 24337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:11.342223 24337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:11.343271 24337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-0/data/instance:
uuid: "e405be04dad64ecfae2d92bb235cd6d3"
format_stamp: "Formatted at 2026-09-28 08:53:11 on dist-test-slave-pbjp"
server_key: "f00f48e8fc25098e194de67a0febecb2"
server_key_iv: "c3af35b9839b4403bb9b2440a3c26cbd"
server_key_version: "encryptionkey@0"
I20260928 08:53:11.343616 24337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance:
uuid: "e405be04dad64ecfae2d92bb235cd6d3"
format_stamp: "Formatted at 2026-09-28 08:53:11 on dist-test-slave-pbjp"
server_key: "f00f48e8fc25098e194de67a0febecb2"
server_key_iv: "c3af35b9839b4403bb9b2440a3c26cbd"
server_key_version: "encryptionkey@0"
I20260928 08:53:11.344857 24337 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:53:11.345659 24352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:11.345811 24337 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:11.345893 24337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "e405be04dad64ecfae2d92bb235cd6d3"
format_stamp: "Formatted at 2026-09-28 08:53:11 on dist-test-slave-pbjp"
server_key: "f00f48e8fc25098e194de67a0febecb2"
server_key_iv: "c3af35b9839b4403bb9b2440a3c26cbd"
server_key_version: "encryptionkey@0"
I20260928 08:53:11.345963 24337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:11.355573 24337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:11.357542 24337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:11.357681 24337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:11.357899 24337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:53:11.358189 24337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:11.358222 24337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:11.358251 24337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:11.358276 24337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:11.363847 24337 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:46183
I20260928 08:53:11.363903 24465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:46183 every 8 connection(s)
I20260928 08:53:11.364194 24337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
I20260928 08:53:11.369563 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 24337
I20260928 08:53:11.369656 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance
I20260928 08:53:11.369812 24223 external_mini_cluster.cc:1428] Setting key da2562c2d60f23a43367cc5025c1c698
Sep 28 08:53:11 dist-test-slave-pbjp krb5kdc[24230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585591, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:11.374083 24333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:11.364840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:40241 (local address 127.23.167.254:33067)
0928 08:53:11.364922 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:11.364925 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:11.365169 (+ 244us) server_negotiation.cc:407] Connection header received
0928 08:53:11.365660 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:11.365661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:11.365671 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:11.365733 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:11.366778 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:11.367285 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:11.368369 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:11.368574 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:11.368628 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:11.371223 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:11.371224 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:11.371225 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:11.371232 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:11.372664 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:11.373145 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:11.373146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:11.373146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:11.373198 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:11.373451 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:11.373452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:11.373453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:11.373561 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:11.373610 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:11.373913 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 08:53:11.373996 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 08:53:11.374310 24468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:11.364828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:33067 (local address 127.23.167.193:40241)
0928 08:53:11.364997 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:11.365046 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:11.365566 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:11.365791 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:11.365795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:11.366481 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:11.366676 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:11.366681 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:11.367482 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:11.367483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:11.368282 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:11.368285 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:11.368640 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:11.368641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:11.368739 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:11.369261 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:11.369270 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:11.371139 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:11.372741 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:11.372749 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:11.372752 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:11.373054 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:11.373254 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:11.373255 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:11.373255 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:11.373384 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:11.373616 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:11.373621 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:11.373786 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 08:53:11.373891 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 08:53:11.374074 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
I20260928 08:53:11.374857 24466 heartbeater.cc:347] Connected to a master server at 127.23.167.254:33067
I20260928 08:53:11.374953 24466 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:11.375123 24466 heartbeater.cc:511] Master 127.23.167.254:33067 requested a full tablet report, sending...
I20260928 08:53:11.375722 24275 ts_manager.cc:194] Registered new tserver with Master: e405be04dad64ecfae2d92bb235cd6d3 (127.23.167.193:46183)
I20260928 08:53:11.376677 24275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.193@KRBTEST.COM'} at 127.23.167.193:40241
WARNING: no policy specified for kudu/127.23.167.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
I20260928 08:53:11.395200 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:0
--local_ip_for_outbound_sockets=127.23.167.194
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:33067
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:35419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:11.473920 24471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:11.474118 24471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:11.474148 24471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:11.475267 24471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:11.475308 24471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:11.475343 24471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:53:11.476745 24471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:35419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:33067
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:11.476966 24471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:11.477198 24471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:11.481910 24476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:11.481948 24479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:11.482002 24471 server_base.cc:1034] running on GCE node
W20260928 08:53:11.481910 24477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:11.482301 24471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:11.482514 24471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:11.483646 24471 hybrid_clock.cc:648] HybridClock initialized: now 1790585591483633 us; error 30 us; skew 500 ppm
Sep 28 08:53:11 dist-test-slave-pbjp krb5kdc[24230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585591, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:11.486047 24471 init.cc:377] Logged in from keytab as kudu/127.23.167.194@KRBTEST.COM (short username kudu)
I20260928 08:53:11.486452 24471 webserver.cc:460] Webserver started at http://127.23.167.194:36041/ using document root <none> and password file <none>
I20260928 08:53:11.486640 24471 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:11.486680 24471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:11.486769 24471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:11.487748 24471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-1/data/instance:
uuid: "3f7ca588ec254b0fa35d1a8160965795"
format_stamp: "Formatted at 2026-09-28 08:53:11 on dist-test-slave-pbjp"
server_key: "9e261f561e775155d3d76ca3a7c19114"
server_key_iv: "0e025629dec307fc349c901a0e841ec9"
server_key_version: "encryptionkey@0"
I20260928 08:53:11.488050 24471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance:
uuid: "3f7ca588ec254b0fa35d1a8160965795"
format_stamp: "Formatted at 2026-09-28 08:53:11 on dist-test-slave-pbjp"
server_key: "9e261f561e775155d3d76ca3a7c19114"
server_key_iv: "0e025629dec307fc349c901a0e841ec9"
server_key_version: "encryptionkey@0"
I20260928 08:53:11.489310 24471 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:53:11.489974 24486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:11.490123 24471 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:11.490195 24471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "3f7ca588ec254b0fa35d1a8160965795"
format_stamp: "Formatted at 2026-09-28 08:53:11 on dist-test-slave-pbjp"
server_key: "9e261f561e775155d3d76ca3a7c19114"
server_key_iv: "0e025629dec307fc349c901a0e841ec9"
server_key_version: "encryptionkey@0"
I20260928 08:53:11.490257 24471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:11.501842 24471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:11.503765 24471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:11.503903 24471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:11.504129 24471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:53:11.504413 24471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:11.504446 24471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:11.504474 24471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:11.504492 24471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:11.510226 24471 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:41873
I20260928 08:53:11.510284 24599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:41873 every 8 connection(s)
I20260928 08:53:11.510594 24471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
Sep 28 08:53:11 dist-test-slave-pbjp krb5kdc[24230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585591, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:11.519526 24333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:11.511362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:58307 (local address 127.23.167.254:33067)
0928 08:53:11.511405 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:11.511407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:11.511584 (+ 177us) server_negotiation.cc:407] Connection header received
0928 08:53:11.512073 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:11.512074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:11.512083 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:11.512143 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:11.513174 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:11.513670 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:11.514664 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:11.514852 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:11.514900 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:11.517036 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:11.517038 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:11.517038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:11.517046 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:11.518377 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:11.518746 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:11.518746 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:11.518746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:11.518792 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:11.518975 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:11.518975 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:11.518975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:11.519078 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:11.519125 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:11.519364 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 08:53:11.519434 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 08:53:11.519760 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 24471
I20260928 08:53:11.519850 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance
I20260928 08:53:11.519995 24223 external_mini_cluster.cc:1428] Setting key b40c357c345d7b7ff9fd46898debbb3e
I20260928 08:53:11.520265 24602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:11.511304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:33067 (local address 127.23.167.194:58307)
0928 08:53:11.511468 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:11.511511 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:11.511991 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:11.512210 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:11.512213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:11.512826 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:11.513035 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:11.513040 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:11.513752 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:11.513753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:11.514591 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:11.514593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:11.514911 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:11.514912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:11.515027 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:11.515523 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:11.515532 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:11.516924 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:11.518454 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:11.518461 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:11.518463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:11.518686 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:11.518838 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:11.518838 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:11.518838 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:11.518934 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:11.519119 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:11.519122 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:11.519256 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 08:53:11.519347 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 08:53:11.519478 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260928 08:53:11.520709 24600 heartbeater.cc:347] Connected to a master server at 127.23.167.254:33067
I20260928 08:53:11.520824 24600 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:11.521014 24600 heartbeater.cc:511] Master 127.23.167.254:33067 requested a full tablet report, sending...
I20260928 08:53:11.521438 24275 ts_manager.cc:194] Registered new tserver with Master: 3f7ca588ec254b0fa35d1a8160965795 (127.23.167.194:41873)
I20260928 08:53:11.521819 24275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.194@KRBTEST.COM'} at 127.23.167.194:58307
WARNING: no policy specified for kudu/127.23.167.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
I20260928 08:53:11.543610 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:0
--local_ip_for_outbound_sockets=127.23.167.195
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:33067
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:35419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:11.623332 24605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:11.623550 24605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:11.623582 24605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:11.624747 24605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:11.624788 24605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:11.624823 24605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:53:11.626319 24605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:35419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:33067
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:11.626562 24605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:11.626778 24605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:11.631652 24611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:11.631659 24610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:11.631659 24613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:11.631866 24605 server_base.cc:1034] running on GCE node
I20260928 08:53:11.632066 24605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:11.632292 24605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:11.633425 24605 hybrid_clock.cc:648] HybridClock initialized: now 1790585591633414 us; error 30 us; skew 500 ppm
Sep 28 08:53:11 dist-test-slave-pbjp krb5kdc[24230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585591, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:11.635823 24605 init.cc:377] Logged in from keytab as kudu/127.23.167.195@KRBTEST.COM (short username kudu)
I20260928 08:53:11.636260 24605 webserver.cc:460] Webserver started at http://127.23.167.195:38411/ using document root <none> and password file <none>
I20260928 08:53:11.636484 24605 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:11.636534 24605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:11.636664 24605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:11.637789 24605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-2/data/instance:
uuid: "c5bfbf4ba6cc4776a6fdb40c2f08633a"
format_stamp: "Formatted at 2026-09-28 08:53:11 on dist-test-slave-pbjp"
server_key: "acc8842c090b06ec298d6fb6205d9464"
server_key_iv: "e6379e443d147c206a51a5c30e80ac53"
server_key_version: "encryptionkey@0"
I20260928 08:53:11.638132 24605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance:
uuid: "c5bfbf4ba6cc4776a6fdb40c2f08633a"
format_stamp: "Formatted at 2026-09-28 08:53:11 on dist-test-slave-pbjp"
server_key: "acc8842c090b06ec298d6fb6205d9464"
server_key_iv: "e6379e443d147c206a51a5c30e80ac53"
server_key_version: "encryptionkey@0"
I20260928 08:53:11.639410 24605 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:53:11.640132 24620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:11.640319 24605 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:53:11.640409 24605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "c5bfbf4ba6cc4776a6fdb40c2f08633a"
format_stamp: "Formatted at 2026-09-28 08:53:11 on dist-test-slave-pbjp"
server_key: "acc8842c090b06ec298d6fb6205d9464"
server_key_iv: "e6379e443d147c206a51a5c30e80ac53"
server_key_version: "encryptionkey@0"
I20260928 08:53:11.640472 24605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:11.653419 24605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:11.655217 24605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:11.655368 24605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:11.655615 24605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:53:11.655923 24605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:11.655956 24605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:11.655987 24605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:11.656010 24605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:11.661880 24605 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:46643
I20260928 08:53:11.661962 24733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:46643 every 8 connection(s)
I20260928 08:53:11.662240 24605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
I20260928 08:53:11.668265 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 24605
I20260928 08:53:11.668375 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance
Sep 28 08:53:11 dist-test-slave-pbjp krb5kdc[24230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585591, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:11.668563 24223 external_mini_cluster.cc:1428] Setting key 86e2ae0623212cc603a7459c0a77be4e
I20260928 08:53:11.671494 24333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:11.662960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:54471 (local address 127.23.167.254:33067)
0928 08:53:11.663019 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:11.663021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:11.663231 (+ 210us) server_negotiation.cc:407] Connection header received
0928 08:53:11.663769 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:11.663770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:11.663782 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:11.663848 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:11.664805 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:11.665366 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:11.666331 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:11.666519 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:11.666568 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:11.668972 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:11.668973 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:11.668973 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:11.668979 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:11.670305 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:11.670736 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:11.670737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:11.670738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:11.670797 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:11.671018 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:11.671018 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:11.671019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:11.671128 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:11.671178 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:11.671375 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 08:53:11.671432 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 08:53:11.671679 24736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:11.662918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:33067 (local address 127.23.167.195:54471)
0928 08:53:11.663089 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:11.663132 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:11.663652 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:11.663917 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:11.663921 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:11.664515 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:11.664730 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:11.664735 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:11.665447 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:11.665448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:11.666259 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:11.666261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:11.666583 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:11.666583 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:11.666697 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:11.667259 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:11.667269 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:11.668844 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:11.670376 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:11.670382 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:11.670383 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:11.670623 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:11.670846 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:11.670847 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:11.670847 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:11.670955 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:11.671169 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:11.671172 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:11.671281 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 08:53:11.671351 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 08:53:11.671490 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260928 08:53:11.672109 24734 heartbeater.cc:347] Connected to a master server at 127.23.167.254:33067
I20260928 08:53:11.672186 24734 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:11.672349 24734 heartbeater.cc:511] Master 127.23.167.254:33067 requested a full tablet report, sending...
I20260928 08:53:11.672678 24275 ts_manager.cc:194] Registered new tserver with Master: c5bfbf4ba6cc4776a6fdb40c2f08633a (127.23.167.195:46643)
I20260928 08:53:11.673026 24275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.195@KRBTEST.COM'} at 127.23.167.195:54471
I20260928 08:53:11.680634 24223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 08:53:11 dist-test-slave-pbjp krb5kdc[24230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585591, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 08:53:11 dist-test-slave-pbjp krb5kdc[24230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585591, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.167.193@KRBTEST.COM
I20260928 08:53:11.703085 24741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:11.695540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48806 (local address 127.23.167.193:46183)
0928 08:53:11.695704 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:11.695708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:11.695718 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:53:11.695743 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:11.695745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:11.695758 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:11.695820 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:11.696337 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:11.697085 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:11.697842 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:11.698304 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:11.698360 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:11.700502 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:11.700509 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:11.700510 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:11.700516 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:11.702026 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:11.702343 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:11.702347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:11.702348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:11.702401 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:11.702559 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:11.702560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:11.702561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:11.702693 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:11.702860 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:11.702903 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 08:53:11.702978 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
W20260928 08:53:11.703451 24380 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48806
Sep 28 08:53:11 dist-test-slave-pbjp krb5kdc[24230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 08:53:11 dist-test-slave-pbjp krb5kdc[24230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.193@KRBTEST.COM
I20260928 08:53:11.724323 24741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:11.717834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48822 (local address 127.23.167.193:46183)
0928 08:53:11.717916 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:11.717918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:11.717933 (+ 15us) server_negotiation.cc:407] Connection header received
0928 08:53:11.717998 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:11.717999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:11.718011 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:11.718059 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:11.718568 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:11.719103 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:11.719802 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:11.720011 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:11.720061 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:11.722154 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:11.722156 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:11.722156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:11.722170 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:11.723441 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:11.723736 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:11.723737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:11.723737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:11.723791 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:11.723913 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:11.723913 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:11.723914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:11.723999 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:11.724081 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:11.724136 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 08:53:11.724195 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 08:53:11.724826 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 24337
I20260928 08:53:11.729425 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 24471
I20260928 08:53:11.734709 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 24605
I20260928 08:53:11.738658 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 24243
2026-09-28T08:53:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1745 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:53:11 dist-test-slave-pbjp krb5kdc[24748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:53:11 dist-test-slave-pbjp krb5kdc[24748](info): set up 2 sockets
Sep 28 08:53:11 dist-test-slave-pbjp krb5kdc[24748](info): commencing operation
krb5kdc: starting...
W20260928 08:53:12.688417 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:53:12 dist-test-slave-pbjp krb5kdc[24748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:53:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:53:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.167.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
I20260928 08:53:12.800041 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:42433
--webserver_interface=127.23.167.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:42433
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:12.879149 24761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:12.879381 24761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:12.879410 24761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:12.880543 24761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:53:12.880584 24761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:12.880600 24761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:12.880617 24761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:53:12.880631 24761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:53:12.882107 24761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:42433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:42433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:12.882350 24761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:12.882560 24761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:12.887069 24769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:12.887138 24766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:12.887116 24761 server_base.cc:1034] running on GCE node
W20260928 08:53:12.887089 24767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:12.887521 24761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:12.887794 24761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:12.888957 24761 hybrid_clock.cc:648] HybridClock initialized: now 1790585592888938 us; error 35 us; skew 500 ppm
Sep 28 08:53:12 dist-test-slave-pbjp krb5kdc[24748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585592, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:12.891228 24761 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:53:12.891655 24761 webserver.cc:460] Webserver started at http://127.23.167.254:35403/ using document root <none> and password file <none>
I20260928 08:53:12.891902 24761 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:12.891968 24761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:12.892099 24761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:12.893298 24761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "0703acd8ea804156b83e6595a0624ca8"
format_stamp: "Formatted at 2026-09-28 08:53:12 on dist-test-slave-pbjp"
server_key: "59b5ccfab7c6b3522668753ac82bfaa4"
server_key_iv: "5c18e983e1067c41a3edca74fb8aca1b"
server_key_version: "encryptionkey@0"
I20260928 08:53:12.893582 24761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "0703acd8ea804156b83e6595a0624ca8"
format_stamp: "Formatted at 2026-09-28 08:53:12 on dist-test-slave-pbjp"
server_key: "59b5ccfab7c6b3522668753ac82bfaa4"
server_key_iv: "5c18e983e1067c41a3edca74fb8aca1b"
server_key_version: "encryptionkey@0"
I20260928 08:53:12.894711 24761 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:53:12.895358 24776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:12.895504 24761 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:53:12.895591 24761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "0703acd8ea804156b83e6595a0624ca8"
format_stamp: "Formatted at 2026-09-28 08:53:12 on dist-test-slave-pbjp"
server_key: "59b5ccfab7c6b3522668753ac82bfaa4"
server_key_iv: "5c18e983e1067c41a3edca74fb8aca1b"
server_key_version: "encryptionkey@0"
I20260928 08:53:12.895653 24761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:12.907169 24761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:12.908864 24761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:12.909000 24761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:12.912667 24761 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:42433
I20260928 08:53:12.912720 24828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:42433 every 8 connection(s)
I20260928 08:53:12.913041 24761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:53:12.913971 24829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:12.914160 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 24761
I20260928 08:53:12.914229 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:53:12.914384 24223 external_mini_cluster.cc:1428] Setting key 739fe6d09dec99780c425f10e201d08e
I20260928 08:53:12.917279 24829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0703acd8ea804156b83e6595a0624ca8: Bootstrap starting.
I20260928 08:53:12.917977 24829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0703acd8ea804156b83e6595a0624ca8: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:12.918241 24829 log.cc:824] T 00000000000000000000000000000000 P 0703acd8ea804156b83e6595a0624ca8: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:12.918903 24829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0703acd8ea804156b83e6595a0624ca8: No bootstrap required, opened a new log
Sep 28 08:53:12 dist-test-slave-pbjp krb5kdc[24748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:12.921521 24829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0703acd8ea804156b83e6595a0624ca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0703acd8ea804156b83e6595a0624ca8" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42433 } }
I20260928 08:53:12.921664 24829 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0703acd8ea804156b83e6595a0624ca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:12.921710 24829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0703acd8ea804156b83e6595a0624ca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0703acd8ea804156b83e6595a0624ca8, State: Initialized, Role: FOLLOWER
I20260928 08:53:12.921837 24829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0703acd8ea804156b83e6595a0624ca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0703acd8ea804156b83e6595a0624ca8" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42433 } }
I20260928 08:53:12.921912 24829 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0703acd8ea804156b83e6595a0624ca8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:53:12.921948 24829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0703acd8ea804156b83e6595a0624ca8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:53:12.921985 24829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0703acd8ea804156b83e6595a0624ca8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:12.922688 24829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0703acd8ea804156b83e6595a0624ca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0703acd8ea804156b83e6595a0624ca8" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42433 } }
I20260928 08:53:12.922801 24829 leader_election.cc:302] T 00000000000000000000000000000000 P 0703acd8ea804156b83e6595a0624ca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0703acd8ea804156b83e6595a0624ca8; no voters:
I20260928 08:53:12.922981 24829 leader_election.cc:288] T 00000000000000000000000000000000 P 0703acd8ea804156b83e6595a0624ca8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:53:12.923046 24834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0703acd8ea804156b83e6595a0624ca8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:53:12.923218 24834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0703acd8ea804156b83e6595a0624ca8 [term 1 LEADER]: Becoming Leader. State: Replica: 0703acd8ea804156b83e6595a0624ca8, State: Running, Role: LEADER
I20260928 08:53:12.923240 24829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0703acd8ea804156b83e6595a0624ca8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:53:12.923316 24834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0703acd8ea804156b83e6595a0624ca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0703acd8ea804156b83e6595a0624ca8" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42433 } }
I20260928 08:53:12.923633 24834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0703acd8ea804156b83e6595a0624ca8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0703acd8ea804156b83e6595a0624ca8. Latest consensus state: current_term: 1 leader_uuid: "0703acd8ea804156b83e6595a0624ca8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0703acd8ea804156b83e6595a0624ca8" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42433 } } }
I20260928 08:53:12.923708 24834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0703acd8ea804156b83e6595a0624ca8 [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:12.923928 24836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0703acd8ea804156b83e6595a0624ca8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0703acd8ea804156b83e6595a0624ca8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0703acd8ea804156b83e6595a0624ca8" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42433 } } }
I20260928 08:53:12.924001 24836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0703acd8ea804156b83e6595a0624ca8 [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:12.923939 24841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:53:12.924556 24841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:53:12.925493 24832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:12.914883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47758 (local address 127.23.167.254:42433)
0928 08:53:12.915080 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:12.915090 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:12.915124 (+ 34us) server_negotiation.cc:407] Connection header received
0928 08:53:12.915572 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:12.915582 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:12.915647 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:12.915801 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:12.916385 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:12.917769 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:12.918558 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:12.919027 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:12.919119 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:12.921645 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:12.921648 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:12.921649 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:12.921660 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:12.923515 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:12.923902 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:12.923905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:12.923906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:12.924024 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:12.924333 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:12.924334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:12.924334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:12.924664 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:12.924801 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:12.925155 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 08:53:12.925295 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":82,"threads_started":1}
I20260928 08:53:12.926062 24841 catalog_manager.cc:1347] Generated new cluster ID: b8917edc10854e5dac720405a251e36e
I20260928 08:53:12.926110 24841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:53:12.943064 24841 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:53:12.943656 24841 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:53:12.954208 24841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0703acd8ea804156b83e6595a0624ca8: Generated new TSK 0
I20260928 08:53:12.954429 24841 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.167.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
I20260928 08:53:12.983539 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:0
--local_ip_for_outbound_sockets=127.23.167.193
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:42433
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:13.063743 24855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:13.063946 24855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:13.063969 24855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:13.065155 24855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:13.065194 24855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:13.065222 24855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:53:13.066694 24855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:42433
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:13.066921 24855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:13.067133 24855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:13.071849 24861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:13.071981 24855 server_base.cc:1034] running on GCE node
W20260928 08:53:13.071877 24863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:13.071871 24860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:13.072317 24855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:13.072573 24855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:13.073707 24855 hybrid_clock.cc:648] HybridClock initialized: now 1790585593073696 us; error 26 us; skew 500 ppm
Sep 28 08:53:13 dist-test-slave-pbjp krb5kdc[24748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585593, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:13.076150 24855 init.cc:377] Logged in from keytab as kudu/127.23.167.193@KRBTEST.COM (short username kudu)
I20260928 08:53:13.076575 24855 webserver.cc:460] Webserver started at http://127.23.167.193:38521/ using document root <none> and password file <none>
I20260928 08:53:13.076766 24855 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:13.076807 24855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:13.076928 24855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:13.077983 24855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-0/data/instance:
uuid: "927a193fd30441efa5c2ff9bec8dea5d"
format_stamp: "Formatted at 2026-09-28 08:53:13 on dist-test-slave-pbjp"
server_key: "29e4567ec74ded6118fc04e0f7581a26"
server_key_iv: "ae881acb8d5a1d73ae5701763faf0c78"
server_key_version: "encryptionkey@0"
I20260928 08:53:13.078294 24855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance:
uuid: "927a193fd30441efa5c2ff9bec8dea5d"
format_stamp: "Formatted at 2026-09-28 08:53:13 on dist-test-slave-pbjp"
server_key: "29e4567ec74ded6118fc04e0f7581a26"
server_key_iv: "ae881acb8d5a1d73ae5701763faf0c78"
server_key_version: "encryptionkey@0"
I20260928 08:53:13.079499 24855 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:53:13.080168 24870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:13.080322 24855 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:53:13.080399 24855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "927a193fd30441efa5c2ff9bec8dea5d"
format_stamp: "Formatted at 2026-09-28 08:53:13 on dist-test-slave-pbjp"
server_key: "29e4567ec74ded6118fc04e0f7581a26"
server_key_iv: "ae881acb8d5a1d73ae5701763faf0c78"
server_key_version: "encryptionkey@0"
I20260928 08:53:13.080457 24855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:13.101933 24855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:13.104059 24855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:13.104171 24855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:13.104355 24855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:53:13.104661 24855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:13.104696 24855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:13.104718 24855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:13.104741 24855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:13.110226 24855 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:37415
I20260928 08:53:13.110294 24983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:37415 every 8 connection(s)
I20260928 08:53:13.110606 24855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
Sep 28 08:53:13 dist-test-slave-pbjp krb5kdc[24748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585593, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:13.118242 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 24855
I20260928 08:53:13.118332 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance
I20260928 08:53:13.118508 24223 external_mini_cluster.cc:1428] Setting key 03ce7c54ed67c74b32d62ecadd72300c
I20260928 08:53:13.120198 24832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:13.111312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:50739 (local address 127.23.167.254:42433)
0928 08:53:13.111434 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:13.111437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:13.111611 (+ 174us) server_negotiation.cc:407] Connection header received
0928 08:53:13.112668 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:13.112670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:13.112684 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:13.112728 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:13.113434 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.113966 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.114969 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.115174 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.115230 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:13.117298 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:13.117299 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:13.117299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:13.117306 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:13.118840 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:13.119298 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:13.119299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:13.119300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:13.119364 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:13.119589 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:13.119590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:13.119590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:13.119728 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:13.119792 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:13.120018 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 08:53:13.120109 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 08:53:13.120270 24986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:13.111277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42433 (local address 127.23.167.193:50739)
0928 08:53:13.111487 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:13.111530 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:13.112551 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:13.112817 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:13.112821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:13.113131 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:13.113334 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:13.113340 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.114055 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.114056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:13.114880 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:13.114883 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.115244 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.115245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:13.115365 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:13.115791 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:13.115799 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:13.117157 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:13.118912 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:13.118919 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:13.118922 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:13.119216 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:13.119440 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:13.119440 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:13.119440 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:13.119539 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:13.119784 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:13.119787 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:13.119901 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 08:53:13.119980 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 08:53:13.120098 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":88,"threads_started":1}
I20260928 08:53:13.120950 24984 heartbeater.cc:347] Connected to a master server at 127.23.167.254:42433
I20260928 08:53:13.121057 24984 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:13.121281 24984 heartbeater.cc:511] Master 127.23.167.254:42433 requested a full tablet report, sending...
I20260928 08:53:13.121788 24793 ts_manager.cc:194] Registered new tserver with Master: 927a193fd30441efa5c2ff9bec8dea5d (127.23.167.193:37415)
I20260928 08:53:13.122382 24793 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.193@KRBTEST.COM'} at 127.23.167.193:50739
WARNING: no policy specified for kudu/127.23.167.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
I20260928 08:53:13.141933 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:0
--local_ip_for_outbound_sockets=127.23.167.194
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:42433
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:13.221297 24989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:13.221493 24989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:13.221517 24989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:13.222616 24989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:13.222652 24989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:13.222678 24989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:53:13.224166 24989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:42433
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:13.224372 24989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:13.224573 24989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:13.229293 24997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:13.229355 24994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:13.229336 24995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:13.229518 24989 server_base.cc:1034] running on GCE node
I20260928 08:53:13.229722 24989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:13.229948 24989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:13.231084 24989 hybrid_clock.cc:648] HybridClock initialized: now 1790585593231066 us; error 32 us; skew 500 ppm
Sep 28 08:53:13 dist-test-slave-pbjp krb5kdc[24748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585593, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:13.233516 24989 init.cc:377] Logged in from keytab as kudu/127.23.167.194@KRBTEST.COM (short username kudu)
I20260928 08:53:13.233970 24989 webserver.cc:460] Webserver started at http://127.23.167.194:33271/ using document root <none> and password file <none>
I20260928 08:53:13.234189 24989 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:13.234236 24989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:13.234359 24989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:13.235452 24989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-1/data/instance:
uuid: "6318793652314c84a23f81c48b35cb1d"
format_stamp: "Formatted at 2026-09-28 08:53:13 on dist-test-slave-pbjp"
server_key: "92b60f13f688696362718736347ce24a"
server_key_iv: "f2325ca23425a0f140b0b591104b743d"
server_key_version: "encryptionkey@0"
I20260928 08:53:13.235808 24989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance:
uuid: "6318793652314c84a23f81c48b35cb1d"
format_stamp: "Formatted at 2026-09-28 08:53:13 on dist-test-slave-pbjp"
server_key: "92b60f13f688696362718736347ce24a"
server_key_iv: "f2325ca23425a0f140b0b591104b743d"
server_key_version: "encryptionkey@0"
I20260928 08:53:13.237025 24989 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:53:13.237718 25004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:13.237879 24989 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:53:13.237958 24989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "6318793652314c84a23f81c48b35cb1d"
format_stamp: "Formatted at 2026-09-28 08:53:13 on dist-test-slave-pbjp"
server_key: "92b60f13f688696362718736347ce24a"
server_key_iv: "f2325ca23425a0f140b0b591104b743d"
server_key_version: "encryptionkey@0"
I20260928 08:53:13.238025 24989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:13.258562 24989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:13.260459 24989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:13.260596 24989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:13.260812 24989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:53:13.261155 24989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:13.261194 24989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:13.261224 24989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:13.261245 24989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:13.266721 24989 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:38081
I20260928 08:53:13.266772 25117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:38081 every 8 connection(s)
I20260928 08:53:13.267089 24989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
Sep 28 08:53:13 dist-test-slave-pbjp krb5kdc[24748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585593, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:13.276348 24832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:13.267836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:57981 (local address 127.23.167.254:42433)
0928 08:53:13.267878 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:13.267881 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:13.268100 (+ 219us) server_negotiation.cc:407] Connection header received
0928 08:53:13.269017 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:13.269019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:13.269030 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:13.269113 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:13.269714 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.270220 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.271293 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.271485 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.271537 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:13.273732 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:13.273733 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:13.273733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:13.273740 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:13.275065 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:13.275469 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:13.275469 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:13.275470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:13.275519 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:13.275745 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:13.275747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:13.275747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:13.275888 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:13.275948 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:13.276208 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 08:53:13.276277 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 08:53:13.276546 25120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:13.267809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42433 (local address 127.23.167.194:57981)
0928 08:53:13.267983 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:13.268023 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:13.268911 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:13.269182 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:13.269185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:13.269444 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:13.269642 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:13.269646 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.270337 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.270338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:13.271207 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:13.271210 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.271547 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.271547 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:13.271647 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:13.272105 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:13.272114 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:13.273623 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:13.275146 (+ 1523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:13.275153 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:13.275155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:13.275410 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:13.275565 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:13.275565 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:13.275566 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:13.275671 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:13.275963 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:13.275969 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:13.276100 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 08:53:13.276190 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 08:53:13.276316 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":74,"threads_started":1}
I20260928 08:53:13.276778 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 24989
I20260928 08:53:13.276846 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance
I20260928 08:53:13.276978 24223 external_mini_cluster.cc:1428] Setting key b89c2539dca24349485bad1c1e56c860
I20260928 08:53:13.277567 25118 heartbeater.cc:347] Connected to a master server at 127.23.167.254:42433
I20260928 08:53:13.277668 25118 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:13.277869 25118 heartbeater.cc:511] Master 127.23.167.254:42433 requested a full tablet report, sending...
I20260928 08:53:13.278254 24792 ts_manager.cc:194] Registered new tserver with Master: 6318793652314c84a23f81c48b35cb1d (127.23.167.194:38081)
I20260928 08:53:13.278631 24792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.194@KRBTEST.COM'} at 127.23.167.194:57981
WARNING: no policy specified for kudu/127.23.167.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
I20260928 08:53:13.299984 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:0
--local_ip_for_outbound_sockets=127.23.167.195
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:42433
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:13.381351 25123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:13.381562 25123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:13.381593 25123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:13.382709 25123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:13.382750 25123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:13.382784 25123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:53:13.384272 25123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:42433
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:13.384500 25123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:13.384701 25123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:13.389546 25129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:13.389616 25128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:13.389843 25131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:13.389964 25123 server_base.cc:1034] running on GCE node
I20260928 08:53:13.390163 25123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:13.390399 25123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:13.391556 25123 hybrid_clock.cc:648] HybridClock initialized: now 1790585593391537 us; error 26 us; skew 500 ppm
Sep 28 08:53:13 dist-test-slave-pbjp krb5kdc[24748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585593, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:13.394572 25123 init.cc:377] Logged in from keytab as kudu/127.23.167.195@KRBTEST.COM (short username kudu)
I20260928 08:53:13.394995 25123 webserver.cc:460] Webserver started at http://127.23.167.195:39105/ using document root <none> and password file <none>
I20260928 08:53:13.395224 25123 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:13.395278 25123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:13.395413 25123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:13.396759 25123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-2/data/instance:
uuid: "8d64a29bb5134e999210750796e53b7c"
format_stamp: "Formatted at 2026-09-28 08:53:13 on dist-test-slave-pbjp"
server_key: "7887ece194026feeb179ab06b1081648"
server_key_iv: "bec2029e9ac38af4b74a360b9d157c58"
server_key_version: "encryptionkey@0"
I20260928 08:53:13.397231 25123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance:
uuid: "8d64a29bb5134e999210750796e53b7c"
format_stamp: "Formatted at 2026-09-28 08:53:13 on dist-test-slave-pbjp"
server_key: "7887ece194026feeb179ab06b1081648"
server_key_iv: "bec2029e9ac38af4b74a360b9d157c58"
server_key_version: "encryptionkey@0"
I20260928 08:53:13.398645 25123 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:53:13.399521 25138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:13.399713 25123 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:53:13.399797 25123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "8d64a29bb5134e999210750796e53b7c"
format_stamp: "Formatted at 2026-09-28 08:53:13 on dist-test-slave-pbjp"
server_key: "7887ece194026feeb179ab06b1081648"
server_key_iv: "bec2029e9ac38af4b74a360b9d157c58"
server_key_version: "encryptionkey@0"
I20260928 08:53:13.399864 25123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:13.413010 25123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:13.415251 25123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:13.415392 25123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:13.415627 25123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:53:13.415928 25123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:13.415962 25123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:13.415983 25123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:13.415994 25123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:13.421517 25123 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:44243
I20260928 08:53:13.421566 25251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:44243 every 8 connection(s)
I20260928 08:53:13.421864 25123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
I20260928 08:53:13.425695 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 25123
I20260928 08:53:13.425784 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance
I20260928 08:53:13.425948 24223 external_mini_cluster.cc:1428] Setting key 52adc6cbbe2845c49b53812c9b223c62
Sep 28 08:53:13 dist-test-slave-pbjp krb5kdc[24748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585593, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:13.431231 24832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:13.422531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:34549 (local address 127.23.167.254:42433)
0928 08:53:13.422618 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:13.422620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:13.422793 (+ 173us) server_negotiation.cc:407] Connection header received
0928 08:53:13.423807 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:13.423808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:13.423821 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:13.423891 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:13.424512 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.425164 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.426436 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.426685 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.426732 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:13.428735 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:13.428736 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:13.428737 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:13.428743 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:13.429960 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:13.430344 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:13.430345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:13.430345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:13.430404 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:13.430671 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:13.430672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:13.430673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:13.430809 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:13.430873 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:13.431093 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 08:53:13.431156 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 08:53:13.431392 25254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:13.422492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42433 (local address 127.23.167.195:34549)
0928 08:53:13.422667 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:13.422710 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:13.423688 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:13.423967 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:13.423971 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:13.424216 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:13.424428 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:13.424433 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.425231 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.425232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:13.426364 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:13.426368 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.426756 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.426757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:13.426904 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:13.427368 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:13.427376 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:13.428657 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:13.430027 (+ 1370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:13.430032 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:13.430035 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:13.430248 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:13.430463 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:13.430464 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:13.430464 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:13.430569 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:13.430870 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:13.430874 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:13.430992 (+ 118us) client_negotiation.cc:769] Sending connection context
0928 08:53:13.431074 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 08:53:13.431199 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
I20260928 08:53:13.431839 25252 heartbeater.cc:347] Connected to a master server at 127.23.167.254:42433
I20260928 08:53:13.431944 25252 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:13.432122 25252 heartbeater.cc:511] Master 127.23.167.254:42433 requested a full tablet report, sending...
I20260928 08:53:13.432554 24792 ts_manager.cc:194] Registered new tserver with Master: 8d64a29bb5134e999210750796e53b7c (127.23.167.195:44243)
I20260928 08:53:13.432916 24792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.195@KRBTEST.COM'} at 127.23.167.195:34549
I20260928 08:53:13.437640 24223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:53:13.445935 24832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:13.439403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47774 (local address 127.23.167.254:42433)
0928 08:53:13.439442 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:13.439444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:13.439587 (+ 143us) server_negotiation.cc:407] Connection header received
0928 08:53:13.439653 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:13.439654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:13.439667 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:13.439724 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:13.440279 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.441120 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.441937 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.442249 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.442315 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:13.443482 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:13.443484 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:13.443484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:13.443492 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:13.444900 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:13.445261 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:13.445262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:13.445262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:13.445313 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:13.445488 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:13.445489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:13.445489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:13.445633 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:13.445706 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:13.445791 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 08:53:13.445866 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 08:53:13.448215 24792 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:53:13.449853 24792 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:53:13.456473 25262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:13.452185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56436 (local address 127.23.167.195:44243)
0928 08:53:13.452917 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:13.452921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:13.452931 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:53:13.452967 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:13.452969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:13.452984 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:13.453050 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:53:13.453300 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.454268 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.455336 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.456051 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.456127 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:13.456186 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:13.456343 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 08:53:13.456385 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":101,"threads_started":1}
I20260928 08:53:13.456485 25264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:13.452307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:37415 (local address 127.0.0.1:39056)
0928 08:53:13.452406 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:13.452446 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:13.452511 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:13.452767 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:13.452771 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:13.452774 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:53:13.452856 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:13.452858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.454162 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.454163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:13.455234 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:13.455236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.456129 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.456130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:13.456224 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:13.456241 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 08:53:13.456318 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 08:53:13.456387 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":5,"thread_start_us":36,"threads_started":1}
I20260928 08:53:13.456485 25261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:13.452107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:44243 (local address 127.0.0.1:56436)
0928 08:53:13.452402 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:13.452446 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:13.452511 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:13.453162 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:13.453163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:13.453164 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:53:13.453217 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:13.453219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.454358 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.454359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:13.455245 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:13.455247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.456131 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.456132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:13.456234 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:13.456241 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 08:53:13.456325 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 08:53:13.456387 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":26,"thread_start_us":60,"threads_started":1}
I20260928 08:53:13.457159 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:13.452436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39056 (local address 127.23.167.193:37415)
0928 08:53:13.452622 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:13.452627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:13.452636 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:53:13.452671 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:13.452672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:13.452684 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:13.452750 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:53:13.452934 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.454061 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.455310 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.456051 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.456127 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:13.456186 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:13.456998 (+ 812us) server_negotiation.cc:299] Negotiation successful
0928 08:53:13.457043 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":79,"threads_started":1}
I20260928 08:53:13.457435 25260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:13.452107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:38081 (local address 127.0.0.1:56432)
0928 08:53:13.452402 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:13.452446 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:13.452511 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:13.452935 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:13.452936 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:13.452937 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:53:13.452975 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:13.452976 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.453941 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.453942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:13.455222 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:13.455225 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.457192 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.457193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:13.457263 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:13.457268 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 08:53:13.457327 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 08:53:13.457367 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":7,"thread_start_us":67,"threads_started":1}
I20260928 08:53:13.457662 25263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:13.452269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56432 (local address 127.23.167.194:38081)
0928 08:53:13.452633 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:13.452637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:13.452644 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:53:13.452674 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:13.452675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:13.452687 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:13.452746 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:53:13.453047 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.453852 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.456221 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.456905 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.456978 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:13.457054 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:13.457549 (+ 495us) server_negotiation.cc:299] Negotiation successful
0928 08:53:13.457593 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":65,"threads_started":1}
I20260928 08:53:13.457621 25186 tablet_service.cc:1467] Processing CreateTablet for tablet d22b0bf2a28d49ad9d9e503245e87c18 (DEFAULT_TABLE table=test-table [id=c7010a713c5744cab9d6d4984d418be3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:53:13.458005 25186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d22b0bf2a28d49ad9d9e503245e87c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:13.457952 24918 tablet_service.cc:1467] Processing CreateTablet for tablet d22b0bf2a28d49ad9d9e503245e87c18 (DEFAULT_TABLE table=test-table [id=c7010a713c5744cab9d6d4984d418be3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:53:13.458235 24918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d22b0bf2a28d49ad9d9e503245e87c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:13.460595 25266 tablet_bootstrap.cc:492] T d22b0bf2a28d49ad9d9e503245e87c18 P 8d64a29bb5134e999210750796e53b7c: Bootstrap starting.
I20260928 08:53:13.460721 25267 tablet_bootstrap.cc:492] T d22b0bf2a28d49ad9d9e503245e87c18 P 927a193fd30441efa5c2ff9bec8dea5d: Bootstrap starting.
I20260928 08:53:13.461392 25267 tablet_bootstrap.cc:654] T d22b0bf2a28d49ad9d9e503245e87c18 P 927a193fd30441efa5c2ff9bec8dea5d: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:13.461421 25266 tablet_bootstrap.cc:654] T d22b0bf2a28d49ad9d9e503245e87c18 P 8d64a29bb5134e999210750796e53b7c: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:13.461376 25052 tablet_service.cc:1467] Processing CreateTablet for tablet d22b0bf2a28d49ad9d9e503245e87c18 (DEFAULT_TABLE table=test-table [id=c7010a713c5744cab9d6d4984d418be3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:53:13.461711 25052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d22b0bf2a28d49ad9d9e503245e87c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:13.461876 25267 log.cc:824] T d22b0bf2a28d49ad9d9e503245e87c18 P 927a193fd30441efa5c2ff9bec8dea5d: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:13.461933 25266 log.cc:824] T d22b0bf2a28d49ad9d9e503245e87c18 P 8d64a29bb5134e999210750796e53b7c: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:13.462625 25267 tablet_bootstrap.cc:492] T d22b0bf2a28d49ad9d9e503245e87c18 P 927a193fd30441efa5c2ff9bec8dea5d: No bootstrap required, opened a new log
I20260928 08:53:13.462698 25267 ts_tablet_manager.cc:1397] T d22b0bf2a28d49ad9d9e503245e87c18 P 927a193fd30441efa5c2ff9bec8dea5d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:53:13.464689 25267 raft_consensus.cc:348] T d22b0bf2a28d49ad9d9e503245e87c18 P 927a193fd30441efa5c2ff9bec8dea5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6318793652314c84a23f81c48b35cb1d" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 38081 } } peers { permanent_uuid: "8d64a29bb5134e999210750796e53b7c" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44243 } } peers { permanent_uuid: "927a193fd30441efa5c2ff9bec8dea5d" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 37415 } }
I20260928 08:53:13.464880 25267 raft_consensus.cc:374] T d22b0bf2a28d49ad9d9e503245e87c18 P 927a193fd30441efa5c2ff9bec8dea5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:13.464920 25267 raft_consensus.cc:729] T d22b0bf2a28d49ad9d9e503245e87c18 P 927a193fd30441efa5c2ff9bec8dea5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927a193fd30441efa5c2ff9bec8dea5d, State: Initialized, Role: FOLLOWER
I20260928 08:53:13.465054 25270 tablet_bootstrap.cc:492] T d22b0bf2a28d49ad9d9e503245e87c18 P 6318793652314c84a23f81c48b35cb1d: Bootstrap starting.
I20260928 08:53:13.465037 25267 consensus_queue.cc:260] T d22b0bf2a28d49ad9d9e503245e87c18 P 927a193fd30441efa5c2ff9bec8dea5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6318793652314c84a23f81c48b35cb1d" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 38081 } } peers { permanent_uuid: "8d64a29bb5134e999210750796e53b7c" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44243 } } peers { permanent_uuid: "927a193fd30441efa5c2ff9bec8dea5d" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 37415 } }
I20260928 08:53:13.465385 25267 ts_tablet_manager.cc:1428] T d22b0bf2a28d49ad9d9e503245e87c18 P 927a193fd30441efa5c2ff9bec8dea5d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:53:13.465499 24984 heartbeater.cc:503] Master 127.23.167.254:42433 was elected leader, sending a full tablet report...
I20260928 08:53:13.465561 25266 tablet_bootstrap.cc:492] T d22b0bf2a28d49ad9d9e503245e87c18 P 8d64a29bb5134e999210750796e53b7c: No bootstrap required, opened a new log
I20260928 08:53:13.465637 25266 ts_tablet_manager.cc:1397] T d22b0bf2a28d49ad9d9e503245e87c18 P 8d64a29bb5134e999210750796e53b7c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 08:53:13.465766 25270 tablet_bootstrap.cc:654] T d22b0bf2a28d49ad9d9e503245e87c18 P 6318793652314c84a23f81c48b35cb1d: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:13.466328 25270 log.cc:824] T d22b0bf2a28d49ad9d9e503245e87c18 P 6318793652314c84a23f81c48b35cb1d: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:13.467080 25270 tablet_bootstrap.cc:492] T d22b0bf2a28d49ad9d9e503245e87c18 P 6318793652314c84a23f81c48b35cb1d: No bootstrap required, opened a new log
I20260928 08:53:13.467150 25270 ts_tablet_manager.cc:1397] T d22b0bf2a28d49ad9d9e503245e87c18 P 6318793652314c84a23f81c48b35cb1d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 08:53:13.467684 25266 raft_consensus.cc:348] T d22b0bf2a28d49ad9d9e503245e87c18 P 8d64a29bb5134e999210750796e53b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6318793652314c84a23f81c48b35cb1d" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 38081 } } peers { permanent_uuid: "8d64a29bb5134e999210750796e53b7c" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44243 } } peers { permanent_uuid: "927a193fd30441efa5c2ff9bec8dea5d" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 37415 } }
I20260928 08:53:13.467864 25266 raft_consensus.cc:374] T d22b0bf2a28d49ad9d9e503245e87c18 P 8d64a29bb5134e999210750796e53b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:13.467900 25266 raft_consensus.cc:729] T d22b0bf2a28d49ad9d9e503245e87c18 P 8d64a29bb5134e999210750796e53b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d64a29bb5134e999210750796e53b7c, State: Initialized, Role: FOLLOWER
I20260928 08:53:13.468015 25266 consensus_queue.cc:260] T d22b0bf2a28d49ad9d9e503245e87c18 P 8d64a29bb5134e999210750796e53b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6318793652314c84a23f81c48b35cb1d" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 38081 } } peers { permanent_uuid: "8d64a29bb5134e999210750796e53b7c" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44243 } } peers { permanent_uuid: "927a193fd30441efa5c2ff9bec8dea5d" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 37415 } }
I20260928 08:53:13.468302 25252 heartbeater.cc:503] Master 127.23.167.254:42433 was elected leader, sending a full tablet report...
I20260928 08:53:13.468305 25266 ts_tablet_manager.cc:1428] T d22b0bf2a28d49ad9d9e503245e87c18 P 8d64a29bb5134e999210750796e53b7c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:53:13.468998 25270 raft_consensus.cc:348] T d22b0bf2a28d49ad9d9e503245e87c18 P 6318793652314c84a23f81c48b35cb1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6318793652314c84a23f81c48b35cb1d" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 38081 } } peers { permanent_uuid: "8d64a29bb5134e999210750796e53b7c" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44243 } } peers { permanent_uuid: "927a193fd30441efa5c2ff9bec8dea5d" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 37415 } }
I20260928 08:53:13.469179 25270 raft_consensus.cc:374] T d22b0bf2a28d49ad9d9e503245e87c18 P 6318793652314c84a23f81c48b35cb1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:13.469223 25270 raft_consensus.cc:729] T d22b0bf2a28d49ad9d9e503245e87c18 P 6318793652314c84a23f81c48b35cb1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6318793652314c84a23f81c48b35cb1d, State: Initialized, Role: FOLLOWER
I20260928 08:53:13.469333 25270 consensus_queue.cc:260] T d22b0bf2a28d49ad9d9e503245e87c18 P 6318793652314c84a23f81c48b35cb1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6318793652314c84a23f81c48b35cb1d" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 38081 } } peers { permanent_uuid: "8d64a29bb5134e999210750796e53b7c" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44243 } } peers { permanent_uuid: "927a193fd30441efa5c2ff9bec8dea5d" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 37415 } }
I20260928 08:53:13.469569 25270 ts_tablet_manager.cc:1428] T d22b0bf2a28d49ad9d9e503245e87c18 P 6318793652314c84a23f81c48b35cb1d: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260928 08:53:13.469615 25118 heartbeater.cc:503] Master 127.23.167.254:42433 was elected leader, sending a full tablet report...
I20260928 08:53:13.476387 25271 raft_consensus.cc:482] T d22b0bf2a28d49ad9d9e503245e87c18 P 927a193fd30441efa5c2ff9bec8dea5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:53:13.476469 25271 raft_consensus.cc:504] T d22b0bf2a28d49ad9d9e503245e87c18 P 927a193fd30441efa5c2ff9bec8dea5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6318793652314c84a23f81c48b35cb1d" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 38081 } } peers { permanent_uuid: "8d64a29bb5134e999210750796e53b7c" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44243 } } peers { permanent_uuid: "927a193fd30441efa5c2ff9bec8dea5d" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 37415 } }
I20260928 08:53:13.476769 25271 leader_election.cc:288] T d22b0bf2a28d49ad9d9e503245e87c18 P 927a193fd30441efa5c2ff9bec8dea5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6318793652314c84a23f81c48b35cb1d (127.23.167.194:38081), 8d64a29bb5134e999210750796e53b7c (127.23.167.195:44243)
I20260928 08:53:13.480000 25263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:13.476995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:34487 (local address 127.23.167.194:38081)
0928 08:53:13.477033 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:13.477035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:13.477058 (+ 23us) server_negotiation.cc:407] Connection header received
0928 08:53:13.477099 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:13.477100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:13.477108 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:13.477151 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:53:13.477318 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.477779 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.479085 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.479682 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.479759 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:13.479768 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:13.479903 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 08:53:13.479940 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 08:53:13.480031 25262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:13.476962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:33821 (local address 127.23.167.195:44243)
0928 08:53:13.476999 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:13.477001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:13.477283 (+ 282us) server_negotiation.cc:407] Connection header received
0928 08:53:13.477348 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:13.477349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:13.477357 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:13.477402 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:53:13.477536 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.478024 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.479205 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.479743 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.479806 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:13.479813 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:13.479945 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 08:53:13.479979 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 08:53:13.480038 24986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:13.476908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:38081 (local address 127.23.167.193:34487)
0928 08:53:13.476965 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:13.476976 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:13.477015 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:13.477170 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:13.477171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:13.477173 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:53:13.477245 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:13.477247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.477850 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.477852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:13.479011 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:13.479014 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.479760 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.479762 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:13.479838 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:13.479843 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 08:53:13.479895 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 08:53:13.479937 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 08:53:13.480052 25275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:13.476909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:44243 (local address 127.23.167.193:33821)
0928 08:53:13.477223 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:13.477233 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:13.477280 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:13.477414 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:13.477414 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:13.477416 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:53:13.477484 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:13.477486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.478083 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.478084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:13.479141 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:13.479143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.479822 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.479823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:13.479883 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:13.479886 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:53:13.479942 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 08:53:13.479971 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":146,"thread_start_us":81,"threads_started":1}
I20260928 08:53:13.480338 25072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d22b0bf2a28d49ad9d9e503245e87c18" candidate_uuid: "927a193fd30441efa5c2ff9bec8dea5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6318793652314c84a23f81c48b35cb1d" is_pre_election: true
I20260928 08:53:13.480376 25206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d22b0bf2a28d49ad9d9e503245e87c18" candidate_uuid: "927a193fd30441efa5c2ff9bec8dea5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d64a29bb5134e999210750796e53b7c" is_pre_election: true
I20260928 08:53:13.480479 25072 raft_consensus.cc:2454] T d22b0bf2a28d49ad9d9e503245e87c18 P 6318793652314c84a23f81c48b35cb1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927a193fd30441efa5c2ff9bec8dea5d in term 0.
I20260928 08:53:13.480484 25206 raft_consensus.cc:2454] T d22b0bf2a28d49ad9d9e503245e87c18 P 8d64a29bb5134e999210750796e53b7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927a193fd30441efa5c2ff9bec8dea5d in term 0.
I20260928 08:53:13.480682 24871 leader_election.cc:302] T d22b0bf2a28d49ad9d9e503245e87c18 P 927a193fd30441efa5c2ff9bec8dea5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6318793652314c84a23f81c48b35cb1d, 927a193fd30441efa5c2ff9bec8dea5d; no voters:
I20260928 08:53:13.480847 25271 raft_consensus.cc:2781] T d22b0bf2a28d49ad9d9e503245e87c18 P 927a193fd30441efa5c2ff9bec8dea5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:53:13.480885 25271 raft_consensus.cc:482] T d22b0bf2a28d49ad9d9e503245e87c18 P 927a193fd30441efa5c2ff9bec8dea5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:53:13.480906 25271 raft_consensus.cc:3037] T d22b0bf2a28d49ad9d9e503245e87c18 P 927a193fd30441efa5c2ff9bec8dea5d [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:13.481575 25271 raft_consensus.cc:504] T d22b0bf2a28d49ad9d9e503245e87c18 P 927a193fd30441efa5c2ff9bec8dea5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6318793652314c84a23f81c48b35cb1d" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 38081 } } peers { permanent_uuid: "8d64a29bb5134e999210750796e53b7c" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44243 } } peers { permanent_uuid: "927a193fd30441efa5c2ff9bec8dea5d" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 37415 } }
I20260928 08:53:13.481701 25271 leader_election.cc:288] T d22b0bf2a28d49ad9d9e503245e87c18 P 927a193fd30441efa5c2ff9bec8dea5d [CANDIDATE]: Term 1 election: Requested vote from peers 6318793652314c84a23f81c48b35cb1d (127.23.167.194:38081), 8d64a29bb5134e999210750796e53b7c (127.23.167.195:44243)
I20260928 08:53:13.481843 25206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d22b0bf2a28d49ad9d9e503245e87c18" candidate_uuid: "927a193fd30441efa5c2ff9bec8dea5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d64a29bb5134e999210750796e53b7c"
I20260928 08:53:13.481863 25072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d22b0bf2a28d49ad9d9e503245e87c18" candidate_uuid: "927a193fd30441efa5c2ff9bec8dea5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6318793652314c84a23f81c48b35cb1d"
I20260928 08:53:13.481911 25206 raft_consensus.cc:3037] T d22b0bf2a28d49ad9d9e503245e87c18 P 8d64a29bb5134e999210750796e53b7c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:13.481920 25072 raft_consensus.cc:3037] T d22b0bf2a28d49ad9d9e503245e87c18 P 6318793652314c84a23f81c48b35cb1d [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:13.482513 25206 raft_consensus.cc:2454] T d22b0bf2a28d49ad9d9e503245e87c18 P 8d64a29bb5134e999210750796e53b7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927a193fd30441efa5c2ff9bec8dea5d in term 1.
I20260928 08:53:13.482625 25072 raft_consensus.cc:2454] T d22b0bf2a28d49ad9d9e503245e87c18 P 6318793652314c84a23f81c48b35cb1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927a193fd30441efa5c2ff9bec8dea5d in term 1.
I20260928 08:53:13.482682 24873 leader_election.cc:302] T d22b0bf2a28d49ad9d9e503245e87c18 P 927a193fd30441efa5c2ff9bec8dea5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d64a29bb5134e999210750796e53b7c, 927a193fd30441efa5c2ff9bec8dea5d; no voters:
I20260928 08:53:13.482764 25271 raft_consensus.cc:2781] T d22b0bf2a28d49ad9d9e503245e87c18 P 927a193fd30441efa5c2ff9bec8dea5d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:53:13.482903 25271 raft_consensus.cc:686] T d22b0bf2a28d49ad9d9e503245e87c18 P 927a193fd30441efa5c2ff9bec8dea5d [term 1 LEADER]: Becoming Leader. State: Replica: 927a193fd30441efa5c2ff9bec8dea5d, State: Running, Role: LEADER
I20260928 08:53:13.482980 25271 consensus_queue.cc:237] T d22b0bf2a28d49ad9d9e503245e87c18 P 927a193fd30441efa5c2ff9bec8dea5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6318793652314c84a23f81c48b35cb1d" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 38081 } } peers { permanent_uuid: "8d64a29bb5134e999210750796e53b7c" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44243 } } peers { permanent_uuid: "927a193fd30441efa5c2ff9bec8dea5d" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 37415 } }
I20260928 08:53:13.483764 24793 catalog_manager.cc:5495] T d22b0bf2a28d49ad9d9e503245e87c18 P 927a193fd30441efa5c2ff9bec8dea5d reported cstate change: term changed from 0 to 1, leader changed from <none> to 927a193fd30441efa5c2ff9bec8dea5d (127.23.167.193). New cstate: current_term: 1 leader_uuid: "927a193fd30441efa5c2ff9bec8dea5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6318793652314c84a23f81c48b35cb1d" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 38081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d64a29bb5134e999210750796e53b7c" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "927a193fd30441efa5c2ff9bec8dea5d" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 37415 } health_report { overall_health: HEALTHY } } }
Sep 28 08:53:13 dist-test-slave-pbjp krb5kdc[24748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.193@KRBTEST.COM
I20260928 08:53:13.491684 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:13.485313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39062 (local address 127.23.167.193:37415)
0928 08:53:13.485369 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:13.485371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:13.485453 (+ 82us) server_negotiation.cc:407] Connection header received
0928 08:53:13.485496 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:13.485497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:13.485505 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:13.485538 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:13.486010 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.486447 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.487087 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.487252 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.487292 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:13.489408 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:13.489416 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:13.489417 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:13.489431 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:13.490866 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:13.491146 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:13.491149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:13.491149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:13.491196 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:13.491312 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:13.491313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:13.491313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:13.491422 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:13.491535 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:13.491560 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 08:53:13.491614 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 08:53:13 dist-test-slave-pbjp krb5kdc[24748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585593, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 08:53:13 dist-test-slave-pbjp krb5kdc[24748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585593, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.167.193@KRBTEST.COM
I20260928 08:53:13.512454 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:13.506368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39076 (local address 127.23.167.193:37415)
0928 08:53:13.506457 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:13.506458 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:13.506467 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:53:13.506518 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:13.506519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:13.506528 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:13.506574 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:13.507058 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.507538 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.508200 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.508380 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.508444 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:13.510497 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:13.510498 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:13.510499 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:13.510505 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:13.511578 (+ 1073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:13.511862 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:13.511863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:13.511863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:13.511906 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:13.512030 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:13.512031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:13.512031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:13.512110 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:13.512194 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:13.512298 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 08:53:13.512357 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 08:53:13.517608 25119 tablet.cc:2369] T d22b0bf2a28d49ad9d9e503245e87c18 P 6318793652314c84a23f81c48b35cb1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 08:53:13 dist-test-slave-pbjp krb5kdc[24748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 08:53:13 dist-test-slave-pbjp krb5kdc[24748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.193@KRBTEST.COM
I20260928 08:53:13.532644 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:13.526450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39088 (local address 127.23.167.193:37415)
0928 08:53:13.526503 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:13.526504 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:13.526542 (+ 38us) server_negotiation.cc:407] Connection header received
0928 08:53:13.526608 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:13.526609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:13.526618 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:13.526668 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:13.527198 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.527671 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.528353 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:13.528534 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:13.528576 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:13.530627 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:13.530629 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:13.530629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:13.530636 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:13.531807 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:13.532095 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:13.532096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:13.532096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:13.532138 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:13.532263 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:13.532264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:13.532264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:13.532344 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:13.532438 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:13.532518 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 08:53:13.532576 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260928 08:53:13.533109 24898 tablet_service.cc:2908] Rejecting scan request for tablet d22b0bf2a28d49ad9d9e503245e87c18: Uninitialized: clean time has not yet been initialized
I20260928 08:53:13.534017 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 24855
I20260928 08:53:13.538543 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 24989
I20260928 08:53:13.542845 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 25123
I20260928 08:53:13.546339 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 24761
2026-09-28T08:53:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1808 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:53:13 dist-test-slave-pbjp krb5kdc[25290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:53:13 dist-test-slave-pbjp krb5kdc[25290](info): set up 2 sockets
Sep 28 08:53:13 dist-test-slave-pbjp krb5kdc[25290](info): commencing operation
krb5kdc: starting...
W20260928 08:53:14.500401 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:53:14 dist-test-slave-pbjp krb5kdc[25290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:53:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:53:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.167.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
I20260928 08:53:14.609695 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:43303
--webserver_interface=127.23.167.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:33203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:43303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:14.688638 25303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:14.688840 25303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:14.688861 25303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:14.690032 25303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:53:14.690071 25303 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 08:53:14.690083 25303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:14.690095 25303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:14.690114 25303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:53:14.690130 25303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:53:14.691542 25303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:33203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:43303
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:43303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:14.691783 25303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:14.691990 25303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:14.696435 25311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:14.696435 25308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:14.696456 25309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:14.696704 25303 server_base.cc:1034] running on GCE node
I20260928 08:53:14.696921 25303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:14.697211 25303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:14.698377 25303 hybrid_clock.cc:648] HybridClock initialized: now 1790585594698362 us; error 29 us; skew 500 ppm
Sep 28 08:53:14 dist-test-slave-pbjp krb5kdc[25290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585594, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:14.714921 25303 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:53:14.715389 25303 webserver.cc:460] Webserver started at http://127.23.167.254:42431/ using document root <none> and password file <none>
I20260928 08:53:14.715648 25303 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:14.715700 25303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:14.715824 25303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:14.716878 25303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "5c3f8ca93a784f60866c3b4c167eda0a"
format_stamp: "Formatted at 2026-09-28 08:53:14 on dist-test-slave-pbjp"
server_key: "77a0882fd45628d8d6c01da8861efc1d"
server_key_iv: "597f752ddd1c65cdf1407d23e36344de"
server_key_version: "encryptionkey@0"
I20260928 08:53:14.717239 25303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "5c3f8ca93a784f60866c3b4c167eda0a"
format_stamp: "Formatted at 2026-09-28 08:53:14 on dist-test-slave-pbjp"
server_key: "77a0882fd45628d8d6c01da8861efc1d"
server_key_iv: "597f752ddd1c65cdf1407d23e36344de"
server_key_version: "encryptionkey@0"
I20260928 08:53:14.718524 25303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:53:14.719195 25318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:14.719393 25303 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:53:14.719504 25303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "5c3f8ca93a784f60866c3b4c167eda0a"
format_stamp: "Formatted at 2026-09-28 08:53:14 on dist-test-slave-pbjp"
server_key: "77a0882fd45628d8d6c01da8861efc1d"
server_key_iv: "597f752ddd1c65cdf1407d23e36344de"
server_key_version: "encryptionkey@0"
I20260928 08:53:14.719566 25303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:14.733108 25303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:14.735026 25303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:14.735174 25303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:14.739773 25303 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:43303
I20260928 08:53:14.739845 25380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:43303 every 8 connection(s)
I20260928 08:53:14.740160 25303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:53:14.740796 25381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:14.743175 25381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c3f8ca93a784f60866c3b4c167eda0a: Bootstrap starting.
I20260928 08:53:14.743773 25381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c3f8ca93a784f60866c3b4c167eda0a: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:14.744037 25381 log.cc:824] T 00000000000000000000000000000000 P 5c3f8ca93a784f60866c3b4c167eda0a: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:14.744246 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 25303
I20260928 08:53:14.744367 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:53:14.744513 24223 external_mini_cluster.cc:1428] Setting key 5d8aa205fe7c02f2fcea3782ac34d637
I20260928 08:53:14.744805 25381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c3f8ca93a784f60866c3b4c167eda0a: No bootstrap required, opened a new log
I20260928 08:53:14.747541 25381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c3f8ca93a784f60866c3b4c167eda0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3f8ca93a784f60866c3b4c167eda0a" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43303 } }
I20260928 08:53:14.747737 25381 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c3f8ca93a784f60866c3b4c167eda0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:14.747828 25381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c3f8ca93a784f60866c3b4c167eda0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c3f8ca93a784f60866c3b4c167eda0a, State: Initialized, Role: FOLLOWER
I20260928 08:53:14.747961 25381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c3f8ca93a784f60866c3b4c167eda0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3f8ca93a784f60866c3b4c167eda0a" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43303 } }
I20260928 08:53:14.748036 25381 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c3f8ca93a784f60866c3b4c167eda0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:53:14.748070 25381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c3f8ca93a784f60866c3b4c167eda0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:53:14.748113 25381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c3f8ca93a784f60866c3b4c167eda0a [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:14.748768 25381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c3f8ca93a784f60866c3b4c167eda0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3f8ca93a784f60866c3b4c167eda0a" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43303 } }
I20260928 08:53:14.748865 25381 leader_election.cc:302] T 00000000000000000000000000000000 P 5c3f8ca93a784f60866c3b4c167eda0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c3f8ca93a784f60866c3b4c167eda0a; no voters:
I20260928 08:53:14.749029 25381 leader_election.cc:288] T 00000000000000000000000000000000 P 5c3f8ca93a784f60866c3b4c167eda0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:53:14.749096 25386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c3f8ca93a784f60866c3b4c167eda0a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:53:14.749236 25386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c3f8ca93a784f60866c3b4c167eda0a [term 1 LEADER]: Becoming Leader. State: Replica: 5c3f8ca93a784f60866c3b4c167eda0a, State: Running, Role: LEADER
I20260928 08:53:14.749341 25381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c3f8ca93a784f60866c3b4c167eda0a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:53:14.749320 25386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c3f8ca93a784f60866c3b4c167eda0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3f8ca93a784f60866c3b4c167eda0a" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43303 } }
I20260928 08:53:14.749679 25386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c3f8ca93a784f60866c3b4c167eda0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c3f8ca93a784f60866c3b4c167eda0a. Latest consensus state: current_term: 1 leader_uuid: "5c3f8ca93a784f60866c3b4c167eda0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3f8ca93a784f60866c3b4c167eda0a" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43303 } } }
I20260928 08:53:14.749756 25386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c3f8ca93a784f60866c3b4c167eda0a [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:14.749940 25387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c3f8ca93a784f60866c3b4c167eda0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c3f8ca93a784f60866c3b4c167eda0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3f8ca93a784f60866c3b4c167eda0a" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43303 } } }
I20260928 08:53:14.750005 25387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c3f8ca93a784f60866c3b4c167eda0a [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:14.750594 25395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:53:14.751124 25395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 08:53:14 dist-test-slave-pbjp krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:14.752624 25395 catalog_manager.cc:1347] Generated new cluster ID: 26af3fa1b7b14d17b574214f75b28fba
I20260928 08:53:14.752673 25395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:53:14.756415 25385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:14.745031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36806 (local address 127.23.167.254:43303)
0928 08:53:14.745257 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:14.745267 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:14.745305 (+ 38us) server_negotiation.cc:407] Connection header received
0928 08:53:14.746124 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:14.746135 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:14.746205 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:14.746371 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:14.746956 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:14.748297 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:14.749136 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:14.749765 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:14.750195 (+ 430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:14.752611 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:14.752615 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:14.752616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:14.752628 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:14.754549 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:14.754941 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:14.754945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:14.754945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:14.755053 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:14.755224 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:14.755225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:14.755225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:14.755543 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:14.755705 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:14.755780 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 08:53:14.755919 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":115,"threads_started":1}
I20260928 08:53:14.763702 25395 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:53:14.764333 25395 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:53:14.772480 25395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c3f8ca93a784f60866c3b4c167eda0a: Generated new TSK 0
I20260928 08:53:14.772657 25395 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.167.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
I20260928 08:53:14.799846 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:0
--local_ip_for_outbound_sockets=127.23.167.193
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:43303
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:33203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:14.880525 25407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:14.880722 25407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:14.880744 25407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:14.881920 25407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:14.881965 25407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:14.881984 25407 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:53:14.882000 25407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:53:14.883378 25407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:33203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.167.254:43303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:14.883595 25407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:14.883801 25407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:14.888463 25415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:14.888439 25413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:14.888456 25412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:14.888542 25407 server_base.cc:1034] running on GCE node
I20260928 08:53:14.888963 25407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:14.889213 25407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:14.890362 25407 hybrid_clock.cc:648] HybridClock initialized: now 1790585594890348 us; error 44 us; skew 500 ppm
Sep 28 08:53:14 dist-test-slave-pbjp krb5kdc[25290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585594, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:14.892848 25407 init.cc:377] Logged in from keytab as kudu/127.23.167.193@KRBTEST.COM (short username kudu)
I20260928 08:53:14.893355 25407 webserver.cc:460] Webserver started at http://127.23.167.193:43589/ using document root <none> and password file <none>
I20260928 08:53:14.893575 25407 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:14.893625 25407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:14.893755 25407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:14.894726 25407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-0/data/instance:
uuid: "7417efc3585546f9a1346f5e37609996"
format_stamp: "Formatted at 2026-09-28 08:53:14 on dist-test-slave-pbjp"
server_key: "a9072f2c1fd1b7c4e65175c667e7a859"
server_key_iv: "e158cae2b8a95fb281d5224f56b79ca3"
server_key_version: "encryptionkey@0"
I20260928 08:53:14.895053 25407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance:
uuid: "7417efc3585546f9a1346f5e37609996"
format_stamp: "Formatted at 2026-09-28 08:53:14 on dist-test-slave-pbjp"
server_key: "a9072f2c1fd1b7c4e65175c667e7a859"
server_key_iv: "e158cae2b8a95fb281d5224f56b79ca3"
server_key_version: "encryptionkey@0"
I20260928 08:53:14.896275 25407 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:53:14.896993 25422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:14.897193 25407 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:53:14.897269 25407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "7417efc3585546f9a1346f5e37609996"
format_stamp: "Formatted at 2026-09-28 08:53:14 on dist-test-slave-pbjp"
server_key: "a9072f2c1fd1b7c4e65175c667e7a859"
server_key_iv: "e158cae2b8a95fb281d5224f56b79ca3"
server_key_version: "encryptionkey@0"
I20260928 08:53:14.897336 25407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:14.903364 25407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:14.905354 25407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:14.905478 25407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:14.905936 25407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:14.905973 25407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:14.906000 25407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:14.906026 25407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:53:14 dist-test-slave-pbjp krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585594, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:14.912572 25407 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:35251
I20260928 08:53:14.912642 25537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:35251 every 8 connection(s)
I20260928 08:53:14.913002 25407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
I20260928 08:53:14.914445 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 25407
I20260928 08:53:14.914539 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance
I20260928 08:53:14.914721 24223 external_mini_cluster.cc:1428] Setting key 832d050635fb9deecc7b5fec4dcd8273
I20260928 08:53:14.915318 25385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:14.906378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:34095 (local address 127.23.167.254:43303)
0928 08:53:14.906426 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:14.906428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:14.906623 (+ 195us) server_negotiation.cc:407] Connection header received
0928 08:53:14.907125 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:14.907127 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:14.907142 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:14.907220 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:14.908084 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:14.908766 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:14.909804 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:14.910009 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:14.910067 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:14.912448 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:14.912450 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:14.912450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:14.912458 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:14.913910 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:14.914369 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:14.914370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:14.914370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:14.914427 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:14.914685 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:14.914686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:14.914686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:14.914810 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:14.914894 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:14.915139 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 08:53:14.915227 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 08:53:14.915478 25431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:14.906345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:43303 (local address 127.23.167.193:34095)
0928 08:53:14.906499 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:14.906534 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:14.907023 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:14.907334 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:14.907339 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:14.907653 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:14.907950 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:14.907957 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:14.908858 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:14.908859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:14.909713 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:14.909716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:14.910100 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:14.910101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:14.910207 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:14.910684 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:14.910692 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:14.912330 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:14.913985 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:14.913992 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:14.913994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:14.914286 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:14.914483 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:14.914484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:14.914485 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:14.914622 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:14.914866 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:14.914873 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:14.915017 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 08:53:14.915108 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 08:53:14.915260 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
I20260928 08:53:14.916038 25538 heartbeater.cc:347] Connected to a master server at 127.23.167.254:43303
I20260928 08:53:14.916122 25538 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:14.916302 25538 heartbeater.cc:511] Master 127.23.167.254:43303 requested a full tablet report, sending...
I20260928 08:53:14.916813 25334 ts_manager.cc:194] Registered new tserver with Master: 7417efc3585546f9a1346f5e37609996 (127.23.167.193:35251)
I20260928 08:53:14.917730 25334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.193@KRBTEST.COM'} at 127.23.167.193:34095
WARNING: no policy specified for kudu/127.23.167.194@KRBTEST.COM; defaulting to no policy
I20260928 08:53:14.923764 25385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:14.918098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:53717 (local address 127.23.167.254:43303)
0928 08:53:14.918141 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:14.918143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:14.918239 (+ 96us) server_negotiation.cc:407] Connection header received
0928 08:53:14.918265 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:14.918266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:14.918276 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:14.918338 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:14.918610 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:14.919312 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:14.919998 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:14.920288 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:14.920362 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:14.921322 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:14.921323 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:14.921323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:14.921330 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:14.922720 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:14.923034 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:14.923036 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:14.923036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:14.923093 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:14.923261 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:14.923261 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:14.923262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:14.923386 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:14.923454 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:14.923638 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 08:53:14.923701 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 08:53:14.923810 25545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:14.918042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:43303 (local address 127.23.167.193:53717)
0928 08:53:14.918178 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:14.918188 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:14.918235 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:14.918342 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:14.918343 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:14.918503 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:14.918554 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:14.918555 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:14.919388 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:14.919388 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:14.919917 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:14.919919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:14.920383 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:14.920384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:14.920457 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:14.920869 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:14.920874 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:14.921246 (+ 372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:14.922813 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:14.922814 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:14.922814 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:14.922970 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:14.923153 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:14.923154 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:14.923154 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:14.923211 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:14.923486 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:14.923487 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:14.923534 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 08:53:14.923605 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 08:53:14.923700 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
I20260928 08:53:14.940052 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:0
--local_ip_for_outbound_sockets=127.23.167.194
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:43303
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:33203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:15.019639 25547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:15.019863 25547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:15.019901 25547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:15.021189 25547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:15.021234 25547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:15.021262 25547 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:53:15.021286 25547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:53:15.023066 25547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:33203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.167.254:43303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:15.023346 25547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:15.023571 25547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:15.028290 25552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:15.028296 25555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:15.028473 25547 server_base.cc:1034] running on GCE node
W20260928 08:53:15.028308 25553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:15.028754 25547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:15.029018 25547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:15.030181 25547 hybrid_clock.cc:648] HybridClock initialized: now 1790585595030167 us; error 27 us; skew 500 ppm
Sep 28 08:53:15 dist-test-slave-pbjp krb5kdc[25290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585595, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:15.032545 25547 init.cc:377] Logged in from keytab as kudu/127.23.167.194@KRBTEST.COM (short username kudu)
I20260928 08:53:15.032940 25547 webserver.cc:460] Webserver started at http://127.23.167.194:36221/ using document root <none> and password file <none>
I20260928 08:53:15.033180 25547 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:15.033232 25547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:15.033352 25547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:15.034395 25547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-1/data/instance:
uuid: "c05be0432c014c318a17e8d389453b91"
format_stamp: "Formatted at 2026-09-28 08:53:15 on dist-test-slave-pbjp"
server_key: "909a599b8159c7a1311368a50fa52ef4"
server_key_iv: "14daa15eba5d49d20bf40b153320c3bd"
server_key_version: "encryptionkey@0"
I20260928 08:53:15.034719 25547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance:
uuid: "c05be0432c014c318a17e8d389453b91"
format_stamp: "Formatted at 2026-09-28 08:53:15 on dist-test-slave-pbjp"
server_key: "909a599b8159c7a1311368a50fa52ef4"
server_key_iv: "14daa15eba5d49d20bf40b153320c3bd"
server_key_version: "encryptionkey@0"
I20260928 08:53:15.035972 25547 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:53:15.036648 25562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:15.036810 25547 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:53:15.036890 25547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "c05be0432c014c318a17e8d389453b91"
format_stamp: "Formatted at 2026-09-28 08:53:15 on dist-test-slave-pbjp"
server_key: "909a599b8159c7a1311368a50fa52ef4"
server_key_iv: "14daa15eba5d49d20bf40b153320c3bd"
server_key_version: "encryptionkey@0"
I20260928 08:53:15.036950 25547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:15.051118 25547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:15.053215 25547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:15.053345 25547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:15.053822 25547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:15.053864 25547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:15.053894 25547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:15.053918 25547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:15.060151 25547 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:36833
I20260928 08:53:15.060214 25677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:36833 every 8 connection(s)
I20260928 08:53:15.060547 25547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
Sep 28 08:53:15 dist-test-slave-pbjp krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585595, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:15.063493 25385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.054309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:53229 (local address 127.23.167.254:43303)
0928 08:53:15.054357 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:15.054360 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:15.054556 (+ 196us) server_negotiation.cc:407] Connection header received
0928 08:53:15.055062 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:15.055064 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:15.055075 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:15.055125 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:15.055878 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.056636 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.058210 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.058496 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.058555 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.061141 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:15.061143 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:15.061143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:15.061149 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:15.062371 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:15.062722 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:15.062723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:15.062723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:15.062769 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:15.063007 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:15.063008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:15.063008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:15.063103 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:15.063148 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:15.063353 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 08:53:15.063408 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 08:53:15.063651 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.054290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:43303 (local address 127.23.167.194:53229)
0928 08:53:15.054442 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:15.054476 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:15.054967 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:15.055198 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:15.055202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:15.055525 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:15.055798 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:15.055804 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.056742 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.056743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:15.058078 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:15.058081 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.058590 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.058591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:15.058743 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.059286 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:15.059298 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:15.061022 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:15.062447 (+ 1425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:15.062453 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:15.062455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:15.062663 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:15.062844 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:15.062845 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:15.062845 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:15.062955 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:15.063147 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:15.063150 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:15.063266 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 08:53:15.063342 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 08:53:15.063460 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20260928 08:53:15.064108 25678 heartbeater.cc:347] Connected to a master server at 127.23.167.254:43303
I20260928 08:53:15.064191 25678 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:15.064365 25678 heartbeater.cc:511] Master 127.23.167.254:43303 requested a full tablet report, sending...
I20260928 08:53:15.064594 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 25547
I20260928 08:53:15.064678 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance
I20260928 08:53:15.064720 25335 ts_manager.cc:194] Registered new tserver with Master: c05be0432c014c318a17e8d389453b91 (127.23.167.194:36833)
I20260928 08:53:15.064872 24223 external_mini_cluster.cc:1428] Setting key bab073b1ab73ed8b1b39428f258f04de
I20260928 08:53:15.065205 25335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.194@KRBTEST.COM'} at 127.23.167.194:53229
I20260928 08:53:15.070950 25385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.065553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:57907 (local address 127.23.167.254:43303)
0928 08:53:15.065589 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:15.065591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:15.065600 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:53:15.065619 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:15.065620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:15.065630 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:15.065681 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:15.066077 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.066718 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.067435 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.067706 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.067778 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.068643 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:15.068645 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:15.068645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:15.068651 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:15.069887 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:15.070247 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:15.070248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:15.070248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:15.070303 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:15.070495 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:15.070495 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:15.070495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:15.070618 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:15.070665 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:15.070819 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 08:53:15.070880 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 08:53:15.070996 25684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.065385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:43303 (local address 127.23.167.194:57907)
0928 08:53:15.065519 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:15.065530 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:15.065580 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:15.065692 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:15.065693 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:15.065928 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:15.065992 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:15.065994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.066814 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.066815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:15.067359 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:15.067361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.067774 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.067774 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:15.067825 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.068214 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:15.068219 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:15.068559 (+ 340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:15.069964 (+ 1405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:15.069966 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:15.069966 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:15.070175 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:15.070359 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:15.070360 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:15.070360 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:15.070440 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:15.070673 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:15.070674 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:15.070724 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 08:53:15.070798 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 08:53:15.070899 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
WARNING: no policy specified for kudu/127.23.167.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
I20260928 08:53:15.089711 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:0
--local_ip_for_outbound_sockets=127.23.167.195
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:43303
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:33203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:15.170559 25687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:15.170783 25687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:15.170807 25687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:15.171998 25687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:15.172041 25687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:15.172062 25687 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:53:15.172078 25687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:53:15.173540 25687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:33203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.167.254:43303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:15.173756 25687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:15.173974 25687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:15.178743 25692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:15.178781 25693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:15.178897 25687 server_base.cc:1034] running on GCE node
W20260928 08:53:15.178743 25695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:15.179169 25687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:15.179428 25687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:15.180574 25687 hybrid_clock.cc:648] HybridClock initialized: now 1790585595180544 us; error 46 us; skew 500 ppm
Sep 28 08:53:15 dist-test-slave-pbjp krb5kdc[25290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585595, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:15.183090 25687 init.cc:377] Logged in from keytab as kudu/127.23.167.195@KRBTEST.COM (short username kudu)
I20260928 08:53:15.183518 25687 webserver.cc:460] Webserver started at http://127.23.167.195:40603/ using document root <none> and password file <none>
I20260928 08:53:15.183709 25687 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:15.183750 25687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:15.183869 25687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:15.184880 25687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-2/data/instance:
uuid: "99fcfc80d4e34d86b21160e36ba1a88b"
format_stamp: "Formatted at 2026-09-28 08:53:15 on dist-test-slave-pbjp"
server_key: "a4032bfb36805d1da059f86f0cb40f02"
server_key_iv: "bbf87a5a076dda297c585acd834d1dfd"
server_key_version: "encryptionkey@0"
I20260928 08:53:15.185230 25687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance:
uuid: "99fcfc80d4e34d86b21160e36ba1a88b"
format_stamp: "Formatted at 2026-09-28 08:53:15 on dist-test-slave-pbjp"
server_key: "a4032bfb36805d1da059f86f0cb40f02"
server_key_iv: "bbf87a5a076dda297c585acd834d1dfd"
server_key_version: "encryptionkey@0"
I20260928 08:53:15.186451 25687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:53:15.187198 25702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:15.187398 25687 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:53:15.187481 25687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "99fcfc80d4e34d86b21160e36ba1a88b"
format_stamp: "Formatted at 2026-09-28 08:53:15 on dist-test-slave-pbjp"
server_key: "a4032bfb36805d1da059f86f0cb40f02"
server_key_iv: "bbf87a5a076dda297c585acd834d1dfd"
server_key_version: "encryptionkey@0"
I20260928 08:53:15.187536 25687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:15.196951 25687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:15.199011 25687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:15.199119 25687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:15.199615 25687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:15.199653 25687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:15.199674 25687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:15.199687 25687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:15.206007 25687 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:44001
I20260928 08:53:15.206115 25817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:44001 every 8 connection(s)
I20260928 08:53:15.206391 25687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
Sep 28 08:53:15 dist-test-slave-pbjp krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585595, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:15.209378 25385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.200022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:40127 (local address 127.23.167.254:43303)
0928 08:53:15.200088 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:15.200091 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:15.200332 (+ 241us) server_negotiation.cc:407] Connection header received
0928 08:53:15.200804 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:15.200806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:15.200821 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:15.200892 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:15.201774 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.202566 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.204055 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.204373 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.204455 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.207076 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:15.207078 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:15.207078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:15.207085 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:15.208205 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:15.208573 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:15.208574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:15.208575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:15.208638 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:15.208830 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:15.208831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:15.208831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:15.208939 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:15.208986 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:15.209247 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 08:53:15.209313 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 08:53:15.209544 25711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.200013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:43303 (local address 127.23.167.195:40127)
0928 08:53:15.200199 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:15.200240 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:15.200708 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:15.200968 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:15.200972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:15.201362 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:15.201652 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:15.201660 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.202668 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.202668 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:15.203962 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:15.203965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.204461 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.204461 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:15.204584 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.205160 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:15.205170 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:15.206932 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:15.208281 (+ 1349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:15.208287 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:15.208289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:15.208506 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:15.208692 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:15.208693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:15.208693 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:15.208785 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:15.208981 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:15.208983 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:15.209138 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 08:53:15.209226 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 08:53:15.209349 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":96,"threads_started":1}
I20260928 08:53:15.210014 25818 heartbeater.cc:347] Connected to a master server at 127.23.167.254:43303
I20260928 08:53:15.210105 25818 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:15.210301 25818 heartbeater.cc:511] Master 127.23.167.254:43303 requested a full tablet report, sending...
I20260928 08:53:15.210637 25334 ts_manager.cc:194] Registered new tserver with Master: 99fcfc80d4e34d86b21160e36ba1a88b (127.23.167.195:44001)
I20260928 08:53:15.211054 25334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.195@KRBTEST.COM'} at 127.23.167.195:40127
I20260928 08:53:15.214496 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 25687
I20260928 08:53:15.214602 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance
I20260928 08:53:15.214777 24223 external_mini_cluster.cc:1428] Setting key 8e2901d11caa77378a73d245269e2528
I20260928 08:53:15.215581 24223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:53:15.216711 25385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.211115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:55371 (local address 127.23.167.254:43303)
0928 08:53:15.211155 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:15.211157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:15.211375 (+ 218us) server_negotiation.cc:407] Connection header received
0928 08:53:15.211484 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:15.211485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:15.211497 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:15.211564 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:15.211829 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.212301 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.212974 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.213209 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.213256 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.214085 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:15.214086 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:15.214086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:15.214091 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:15.215631 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:15.215958 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:15.215959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:15.215959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:15.216027 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:15.216217 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:15.216217 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:15.216218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:15.216343 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:15.216399 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:15.216539 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 08:53:15.216614 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 08:53:15.216715 25824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.211081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:43303 (local address 127.23.167.195:55371)
0928 08:53:15.211325 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:15.211335 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:15.211431 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:15.211566 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:15.211567 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:15.211718 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:15.211769 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:15.211771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.212414 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.212415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:15.212916 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:15.212917 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.213261 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.213261 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:15.213313 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.213743 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:15.213748 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:15.214024 (+ 276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:15.215712 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:15.215713 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:15.215714 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:15.215897 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:15.216080 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:15.216080 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:15.216081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:15.216162 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:15.216397 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:15.216398 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:15.216447 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 08:53:15.216507 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 08:53:15.216617 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":178,"threads_started":1}
Sep 28 08:53:15 dist-test-slave-pbjp krb5kdc[25290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585595, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 08:53:15 dist-test-slave-pbjp krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585595, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:15.237447 25385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.230981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36812 (local address 127.23.167.254:43303)
0928 08:53:15.231040 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:15.231042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:15.231050 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:53:15.231076 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:15.231077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:15.231086 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:15.231135 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:15.231693 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.232189 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.232942 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.233171 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.233222 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.235376 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:15.235377 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:15.235378 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:15.235385 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:15.236582 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:15.236862 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:15.236863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:15.236863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:15.236907 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:15.237035 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:15.237036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:15.237036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:15.237159 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:15.237230 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:15.237319 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 08:53:15.237373 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 08:53:15.238833 25334 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:53:15.240486 25334 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:53:15.246647 25837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.242917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47434 (local address 127.23.167.195:44001)
0928 08:53:15.243089 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:15.243094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:15.243102 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:53:15.243135 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:15.243136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:15.243147 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:15.243214 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:53:15.243396 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.244169 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.245771 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.246254 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.246328 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.246389 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:15.246517 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 08:53:15.246554 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":82,"threads_started":1}
I20260928 08:53:15.246636 25836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.242701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:44001 (local address 127.0.0.1:47434)
0928 08:53:15.242933 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:15.242942 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:15.242988 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:15.243258 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:15.243260 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:15.243262 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:53:15.243319 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:15.243320 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.244242 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.244243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:15.245512 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:15.245515 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.246339 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.246340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:15.246417 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.246432 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 08:53:15.246498 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 08:53:15.246555 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":1,"thread_start_us":88,"threads_started":1}
I20260928 08:53:15.246773 25834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.242602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54590 (local address 127.23.167.194:36833)
0928 08:53:15.242792 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:15.242796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:15.242905 (+ 109us) server_negotiation.cc:407] Connection header received
0928 08:53:15.243018 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:15.243020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:15.243034 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:15.243078 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:53:15.243313 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.244466 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.245587 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.246315 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.246389 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.246441 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:15.246632 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 08:53:15.246674 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":88,"threads_started":1}
I20260928 08:53:15.246820 25833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.242486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:35251 (local address 127.0.0.1:37306)
0928 08:53:15.242623 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:15.242664 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:15.242722 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:15.243131 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:15.243134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:15.243137 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:53:15.243228 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:15.243231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.244549 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.244549 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:15.245695 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:15.245697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.246623 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.246624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:15.246676 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.246679 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:53:15.246720 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 08:53:15.246758 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":31,"thread_start_us":88,"threads_started":1}
I20260928 08:53:15.246960 25832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.242487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:36833 (local address 127.0.0.1:54590)
0928 08:53:15.242839 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:15.242849 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:15.242931 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:15.243164 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:15.243164 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:15.243166 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:53:15.243228 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:15.243231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.244567 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.244567 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:15.245513 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:15.245515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.246480 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.246481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:15.246540 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.246544 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:53:15.246857 (+ 313us) client_negotiation.cc:240] Negotiation successful
0928 08:53:15.246895 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":12,"thread_start_us":67,"threads_started":1}
I20260928 08:53:15.247030 25835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.242676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37306 (local address 127.23.167.193:35251)
0928 08:53:15.242865 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:15.242868 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:15.242878 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:53:15.242918 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:15.242932 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:15.242949 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:15.243022 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:53:15.243309 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.244465 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.245768 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.246487 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.246546 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.246594 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:15.246894 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 08:53:15.246938 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":78,"threads_started":1}
I20260928 08:53:15.247759 25612 tablet_service.cc:1467] Processing CreateTablet for tablet c09a8aa14cbf4e37b0cd60c52e18c8a8 (DEFAULT_TABLE table=test-table [id=4e5ea306145c441f82c41292bad2c5fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:53:15.247875 25752 tablet_service.cc:1467] Processing CreateTablet for tablet c09a8aa14cbf4e37b0cd60c52e18c8a8 (DEFAULT_TABLE table=test-table [id=4e5ea306145c441f82c41292bad2c5fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:53:15.248073 25612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c09a8aa14cbf4e37b0cd60c52e18c8a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:15.248214 25752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c09a8aa14cbf4e37b0cd60c52e18c8a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:15.249569 25472 tablet_service.cc:1467] Processing CreateTablet for tablet c09a8aa14cbf4e37b0cd60c52e18c8a8 (DEFAULT_TABLE table=test-table [id=4e5ea306145c441f82c41292bad2c5fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:53:15.249963 25472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c09a8aa14cbf4e37b0cd60c52e18c8a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:15.251139 25838 tablet_bootstrap.cc:492] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b: Bootstrap starting.
I20260928 08:53:15.251982 25838 tablet_bootstrap.cc:654] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:15.252213 25838 log.cc:824] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:15.252489 25839 tablet_bootstrap.cc:492] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P c05be0432c014c318a17e8d389453b91: Bootstrap starting.
I20260928 08:53:15.252946 25838 tablet_bootstrap.cc:492] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b: No bootstrap required, opened a new log
I20260928 08:53:15.253006 25838 ts_tablet_manager.cc:1397] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:53:15.253005 25841 tablet_bootstrap.cc:492] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 7417efc3585546f9a1346f5e37609996: Bootstrap starting.
I20260928 08:53:15.253283 25839 tablet_bootstrap.cc:654] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P c05be0432c014c318a17e8d389453b91: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:15.253590 25839 log.cc:824] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P c05be0432c014c318a17e8d389453b91: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:15.253646 25841 tablet_bootstrap.cc:654] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 7417efc3585546f9a1346f5e37609996: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:15.253928 25841 log.cc:824] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 7417efc3585546f9a1346f5e37609996: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:15.254343 25839 tablet_bootstrap.cc:492] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P c05be0432c014c318a17e8d389453b91: No bootstrap required, opened a new log
I20260928 08:53:15.254395 25839 ts_tablet_manager.cc:1397] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P c05be0432c014c318a17e8d389453b91: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:53:15.254756 25838 raft_consensus.cc:348] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } } peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } }
I20260928 08:53:15.254901 25838 raft_consensus.cc:374] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:15.254930 25838 raft_consensus.cc:729] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99fcfc80d4e34d86b21160e36ba1a88b, State: Initialized, Role: FOLLOWER
I20260928 08:53:15.255049 25838 consensus_queue.cc:260] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } } peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } }
I20260928 08:53:15.255286 25838 ts_tablet_manager.cc:1428] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 08:53:15.255354 25818 heartbeater.cc:503] Master 127.23.167.254:43303 was elected leader, sending a full tablet report...
I20260928 08:53:15.256136 25839 raft_consensus.cc:348] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P c05be0432c014c318a17e8d389453b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } } peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } }
I20260928 08:53:15.256278 25839 raft_consensus.cc:374] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P c05be0432c014c318a17e8d389453b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:15.256317 25839 raft_consensus.cc:729] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P c05be0432c014c318a17e8d389453b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05be0432c014c318a17e8d389453b91, State: Initialized, Role: FOLLOWER
I20260928 08:53:15.256428 25839 consensus_queue.cc:260] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P c05be0432c014c318a17e8d389453b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } } peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } }
I20260928 08:53:15.256708 25839 ts_tablet_manager.cc:1428] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P c05be0432c014c318a17e8d389453b91: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 08:53:15.256765 25678 heartbeater.cc:503] Master 127.23.167.254:43303 was elected leader, sending a full tablet report...
I20260928 08:53:15.257462 25841 tablet_bootstrap.cc:492] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 7417efc3585546f9a1346f5e37609996: No bootstrap required, opened a new log
I20260928 08:53:15.257539 25841 ts_tablet_manager.cc:1397] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 7417efc3585546f9a1346f5e37609996: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 08:53:15.259069 25841 raft_consensus.cc:348] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 7417efc3585546f9a1346f5e37609996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } } peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } }
I20260928 08:53:15.259200 25841 raft_consensus.cc:374] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 7417efc3585546f9a1346f5e37609996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:15.259231 25841 raft_consensus.cc:729] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 7417efc3585546f9a1346f5e37609996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7417efc3585546f9a1346f5e37609996, State: Initialized, Role: FOLLOWER
I20260928 08:53:15.259372 25841 consensus_queue.cc:260] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 7417efc3585546f9a1346f5e37609996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } } peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } }
I20260928 08:53:15.259591 25841 ts_tablet_manager.cc:1428] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 7417efc3585546f9a1346f5e37609996: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:53:15.259650 25538 heartbeater.cc:503] Master 127.23.167.254:43303 was elected leader, sending a full tablet report...
I20260928 08:53:15.305212 25844 raft_consensus.cc:482] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:53:15.305365 25844 raft_consensus.cc:504] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } } peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } }
I20260928 08:53:15.305747 25844 leader_election.cc:288] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c05be0432c014c318a17e8d389453b91 (127.23.167.194:36833), 7417efc3585546f9a1346f5e37609996 (127.23.167.193:35251)
I20260928 08:53:15.309278 25711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.306006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:36833 (local address 127.23.167.195:48021)
0928 08:53:15.306047 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:15.306059 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:15.306122 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:15.306315 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:15.306316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:15.306318 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:53:15.306410 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:15.306412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.307250 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.307251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:15.308386 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:15.308389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.308977 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.308978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:15.309100 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.309104 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:53:15.309161 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 08:53:15.309199 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18,"mutex_wait_us":5}
I20260928 08:53:15.309280 25834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.306083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:48021 (local address 127.23.167.194:36833)
0928 08:53:15.306130 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:15.306132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:15.306173 (+ 41us) server_negotiation.cc:407] Connection header received
0928 08:53:15.306204 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:15.306205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:15.306215 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:15.306260 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:53:15.306485 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.307140 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.308489 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.308902 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.308969 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.308979 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:15.309168 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 08:53:15.309209 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 08:53:15.309644 25632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09a8aa14cbf4e37b0cd60c52e18c8a8" candidate_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05be0432c014c318a17e8d389453b91" is_pre_election: true
I20260928 08:53:15.309823 25632 raft_consensus.cc:2454] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P c05be0432c014c318a17e8d389453b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99fcfc80d4e34d86b21160e36ba1a88b in term 0.
I20260928 08:53:15.309890 25835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.306616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:55381 (local address 127.23.167.193:35251)
0928 08:53:15.306881 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:15.306883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:15.306893 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:53:15.306933 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:15.306934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:15.306942 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:15.306995 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:53:15.307178 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.307882 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.309048 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.309492 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.309570 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.309578 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:15.309778 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 08:53:15.309824 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237}
I20260928 08:53:15.309897 25847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.306219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:35251 (local address 127.23.167.195:55381)
0928 08:53:15.306418 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:15.306429 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:15.306477 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:15.307046 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:15.307047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:15.307048 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:53:15.307111 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:15.307113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.307978 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.307979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:15.308965 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:15.308968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.309597 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.309598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:15.309667 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.309672 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 08:53:15.309751 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 08:53:15.309788 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":124,"threads_started":1}
I20260928 08:53:15.310091 25705 leader_election.cc:302] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99fcfc80d4e34d86b21160e36ba1a88b, c05be0432c014c318a17e8d389453b91; no voters:
I20260928 08:53:15.310213 25844 raft_consensus.cc:2781] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:53:15.310267 25844 raft_consensus.cc:482] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:53:15.310290 25844 raft_consensus.cc:3037] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:15.310281 25492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09a8aa14cbf4e37b0cd60c52e18c8a8" candidate_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7417efc3585546f9a1346f5e37609996" is_pre_election: true
I20260928 08:53:15.310444 25492 raft_consensus.cc:2454] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 7417efc3585546f9a1346f5e37609996 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99fcfc80d4e34d86b21160e36ba1a88b in term 0.
W20260928 08:53:15.311190 25679 tablet.cc:2369] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P c05be0432c014c318a17e8d389453b91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:53:15.311311 25844 raft_consensus.cc:504] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } } peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } }
I20260928 08:53:15.311460 25844 leader_election.cc:288] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b [CANDIDATE]: Term 1 election: Requested vote from peers c05be0432c014c318a17e8d389453b91 (127.23.167.194:36833), 7417efc3585546f9a1346f5e37609996 (127.23.167.193:35251)
I20260928 08:53:15.311583 25492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09a8aa14cbf4e37b0cd60c52e18c8a8" candidate_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7417efc3585546f9a1346f5e37609996"
I20260928 08:53:15.311655 25492 raft_consensus.cc:3037] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 7417efc3585546f9a1346f5e37609996 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:15.312415 25492 raft_consensus.cc:2454] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 7417efc3585546f9a1346f5e37609996 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99fcfc80d4e34d86b21160e36ba1a88b in term 1.
I20260928 08:53:15.312521 25632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09a8aa14cbf4e37b0cd60c52e18c8a8" candidate_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05be0432c014c318a17e8d389453b91"
I20260928 08:53:15.312608 25706 leader_election.cc:302] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7417efc3585546f9a1346f5e37609996, 99fcfc80d4e34d86b21160e36ba1a88b; no voters:
I20260928 08:53:15.312602 25632 raft_consensus.cc:3037] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P c05be0432c014c318a17e8d389453b91 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:15.313339 25632 raft_consensus.cc:2454] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P c05be0432c014c318a17e8d389453b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99fcfc80d4e34d86b21160e36ba1a88b in term 1.
I20260928 08:53:15.313434 25844 raft_consensus.cc:2781] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:53:15.313594 25844 raft_consensus.cc:686] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b [term 1 LEADER]: Becoming Leader. State: Replica: 99fcfc80d4e34d86b21160e36ba1a88b, State: Running, Role: LEADER
I20260928 08:53:15.313687 25844 consensus_queue.cc:237] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } } peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } }
I20260928 08:53:15.314648 25334 catalog_manager.cc:5495] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99fcfc80d4e34d86b21160e36ba1a88b (127.23.167.195). New cstate: current_term: 1 leader_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } health_report { overall_health: HEALTHY } } }
I20260928 08:53:15.322623 25837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.320156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47444 (local address 127.23.167.195:44001)
0928 08:53:15.320201 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:15.320203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:15.320212 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:53:15.320289 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:15.320290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:15.320310 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:15.320361 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:53:15.320490 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.321017 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.321862 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.322055 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.322109 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.322307 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:53:15.322401 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:53:15.322486 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:15.322530 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 08:53:15.322553 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 08:53:15.324368 25492 raft_consensus.cc:1261] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 7417efc3585546f9a1346f5e37609996 [term 1 FOLLOWER]: Refusing update from remote peer 99fcfc80d4e34d86b21160e36ba1a88b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:53:15.324688 25632 raft_consensus.cc:1261] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P c05be0432c014c318a17e8d389453b91 [term 1 FOLLOWER]: Refusing update from remote peer 99fcfc80d4e34d86b21160e36ba1a88b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:53:15.324710 25848 consensus_queue.cc:1035] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:53:15.324949 25848 consensus_queue.cc:1035] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:53:15.327025 25851 mvcc.cc:205] Tried to move back new op lower bound from 7334238598445932544 to 7334238598405480448. Current Snapshot: MvccSnapshot[applied={T|T < 7334238598445932544}]
I20260928 08:53:15.327160 25853 mvcc.cc:205] Tried to move back new op lower bound from 7334238598445932544 to 7334238598405480448. Current Snapshot: MvccSnapshot[applied={T|T < 7334238598445932544}]
I20260928 08:53:15.330355 25335 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:53:15.330456 25335 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:53:15.331295 25335 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c3f8ca93a784f60866c3b4c167eda0a: Sending DeleteTablet for 3 replicas of tablet c09a8aa14cbf4e37b0cd60c52e18c8a8
I20260928 08:53:15.331593 25472 tablet_service.cc:1514] Processing DeleteTablet for tablet c09a8aa14cbf4e37b0cd60c52e18c8a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:53:15 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:37306
I20260928 08:53:15.331611 25752 tablet_service.cc:1514] Processing DeleteTablet for tablet c09a8aa14cbf4e37b0cd60c52e18c8a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:53:15 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:47434
I20260928 08:53:15.331830 25862 tablet_replica.cc:330] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b: stopping tablet replica
I20260928 08:53:15.331820 25612 tablet_service.cc:1514] Processing DeleteTablet for tablet c09a8aa14cbf4e37b0cd60c52e18c8a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:53:15 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:54590
I20260928 08:53:15.332000 25862 raft_consensus.cc:2229] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:53:15.332032 25861 tablet_replica.cc:330] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 7417efc3585546f9a1346f5e37609996: stopping tablet replica
I20260928 08:53:15.332125 25861 raft_consensus.cc:2229] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 7417efc3585546f9a1346f5e37609996 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:53:15.332238 25861 raft_consensus.cc:2258] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 7417efc3585546f9a1346f5e37609996 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:53:15.332252 25862 raft_consensus.cc:2258] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:53:15.332274 25335 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:53:15.332417 25335 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:53:15.332706 25861 ts_tablet_manager.cc:1905] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 7417efc3585546f9a1346f5e37609996: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:53:15.332983 25862 ts_tablet_manager.cc:1905] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:53:15.333673 25472 tablet_service.cc:1467] Processing CreateTablet for tablet 0d5ceed395c54ecb84e8c236d239e361 (DEFAULT_TABLE table=test-table [id=620942273acc4683b832164bf6b36ecf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:53:15.333814 25472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d5ceed395c54ecb84e8c236d239e361. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:15.334491 25861 ts_tablet_manager.cc:1918] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 7417efc3585546f9a1346f5e37609996: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:53:15.334569 25861 log.cc:1196] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 7417efc3585546f9a1346f5e37609996: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-0/wal/wals/c09a8aa14cbf4e37b0cd60c52e18c8a8
I20260928 08:53:15.334826 25861 ts_tablet_manager.cc:1939] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 7417efc3585546f9a1346f5e37609996: Deleting consensus metadata
I20260928 08:53:15.334823 25862 ts_tablet_manager.cc:1918] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:53:15.334978 25862 log.cc:1196] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-2/wal/wals/c09a8aa14cbf4e37b0cd60c52e18c8a8
I20260928 08:53:15.335121 25841 tablet_bootstrap.cc:492] T 0d5ceed395c54ecb84e8c236d239e361 P 7417efc3585546f9a1346f5e37609996: Bootstrap starting.
I20260928 08:53:15.335258 25862 ts_tablet_manager.cc:1939] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P 99fcfc80d4e34d86b21160e36ba1a88b: Deleting consensus metadata
I20260928 08:53:15.335537 25841 tablet_bootstrap.cc:654] T 0d5ceed395c54ecb84e8c236d239e361 P 7417efc3585546f9a1346f5e37609996: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:15.335652 25322 catalog_manager.cc:4841] TS 99fcfc80d4e34d86b21160e36ba1a88b (127.23.167.195:44001): tablet c09a8aa14cbf4e37b0cd60c52e18c8a8 (table test-table [id=4e5ea306145c441f82c41292bad2c5fb]) successfully deleted
I20260928 08:53:15.335865 25322 catalog_manager.cc:4841] TS 7417efc3585546f9a1346f5e37609996 (127.23.167.193:35251): tablet c09a8aa14cbf4e37b0cd60c52e18c8a8 (table test-table [id=4e5ea306145c441f82c41292bad2c5fb]) successfully deleted
I20260928 08:53:15.336054 25752 tablet_service.cc:1467] Processing CreateTablet for tablet 0d5ceed395c54ecb84e8c236d239e361 (DEFAULT_TABLE table=test-table [id=620942273acc4683b832164bf6b36ecf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:53:15.336158 25752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d5ceed395c54ecb84e8c236d239e361. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:15.336895 25838 tablet_bootstrap.cc:492] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b: Bootstrap starting.
I20260928 08:53:15.336998 25612 tablet_service.cc:1467] Processing CreateTablet for tablet 0d5ceed395c54ecb84e8c236d239e361 (DEFAULT_TABLE table=test-table [id=620942273acc4683b832164bf6b36ecf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:53:15.337126 25612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d5ceed395c54ecb84e8c236d239e361. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:15.337276 25838 tablet_bootstrap.cc:654] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:15.337919 25838 tablet_bootstrap.cc:492] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b: No bootstrap required, opened a new log
I20260928 08:53:15.337975 25838 ts_tablet_manager.cc:1397] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 08:53:15.338122 25838 raft_consensus.cc:348] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } }
I20260928 08:53:15.338191 25838 raft_consensus.cc:374] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:15.338208 25838 raft_consensus.cc:729] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99fcfc80d4e34d86b21160e36ba1a88b, State: Initialized, Role: FOLLOWER
I20260928 08:53:15.338251 25838 consensus_queue.cc:260] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } }
I20260928 08:53:15.338331 25838 ts_tablet_manager.cc:1428] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 08:53:15.338526 25839 tablet_bootstrap.cc:492] T 0d5ceed395c54ecb84e8c236d239e361 P c05be0432c014c318a17e8d389453b91: Bootstrap starting.
I20260928 08:53:15.338883 25839 tablet_bootstrap.cc:654] T 0d5ceed395c54ecb84e8c236d239e361 P c05be0432c014c318a17e8d389453b91: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:15.339301 25841 tablet_bootstrap.cc:492] T 0d5ceed395c54ecb84e8c236d239e361 P 7417efc3585546f9a1346f5e37609996: No bootstrap required, opened a new log
I20260928 08:53:15.339347 25841 ts_tablet_manager.cc:1397] T 0d5ceed395c54ecb84e8c236d239e361 P 7417efc3585546f9a1346f5e37609996: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 08:53:15.339476 25839 tablet_bootstrap.cc:492] T 0d5ceed395c54ecb84e8c236d239e361 P c05be0432c014c318a17e8d389453b91: No bootstrap required, opened a new log
I20260928 08:53:15.339469 25841 raft_consensus.cc:348] T 0d5ceed395c54ecb84e8c236d239e361 P 7417efc3585546f9a1346f5e37609996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } }
I20260928 08:53:15.339514 25841 raft_consensus.cc:374] T 0d5ceed395c54ecb84e8c236d239e361 P 7417efc3585546f9a1346f5e37609996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:15.339526 25841 raft_consensus.cc:729] T 0d5ceed395c54ecb84e8c236d239e361 P 7417efc3585546f9a1346f5e37609996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7417efc3585546f9a1346f5e37609996, State: Initialized, Role: FOLLOWER
I20260928 08:53:15.339517 25839 ts_tablet_manager.cc:1397] T 0d5ceed395c54ecb84e8c236d239e361 P c05be0432c014c318a17e8d389453b91: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 08:53:15.339560 25841 consensus_queue.cc:260] T 0d5ceed395c54ecb84e8c236d239e361 P 7417efc3585546f9a1346f5e37609996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } }
I20260928 08:53:15.339633 25841 ts_tablet_manager.cc:1428] T 0d5ceed395c54ecb84e8c236d239e361 P 7417efc3585546f9a1346f5e37609996: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:15.339659 25839 raft_consensus.cc:348] T 0d5ceed395c54ecb84e8c236d239e361 P c05be0432c014c318a17e8d389453b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } }
I20260928 08:53:15.339725 25839 raft_consensus.cc:374] T 0d5ceed395c54ecb84e8c236d239e361 P c05be0432c014c318a17e8d389453b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:15.339744 25839 raft_consensus.cc:729] T 0d5ceed395c54ecb84e8c236d239e361 P c05be0432c014c318a17e8d389453b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05be0432c014c318a17e8d389453b91, State: Initialized, Role: FOLLOWER
I20260928 08:53:15.339771 25839 consensus_queue.cc:260] T 0d5ceed395c54ecb84e8c236d239e361 P c05be0432c014c318a17e8d389453b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } }
I20260928 08:53:15.339852 25839 ts_tablet_manager.cc:1428] T 0d5ceed395c54ecb84e8c236d239e361 P c05be0432c014c318a17e8d389453b91: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:15.340111 25863 tablet_replica.cc:330] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P c05be0432c014c318a17e8d389453b91: stopping tablet replica
I20260928 08:53:15.340219 25863 raft_consensus.cc:2229] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P c05be0432c014c318a17e8d389453b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:53:15.340332 25863 raft_consensus.cc:2258] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P c05be0432c014c318a17e8d389453b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:53:15.340674 25863 ts_tablet_manager.cc:1905] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P c05be0432c014c318a17e8d389453b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:53:15.342085 25863 ts_tablet_manager.cc:1918] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P c05be0432c014c318a17e8d389453b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:53:15.342139 25863 log.cc:1196] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P c05be0432c014c318a17e8d389453b91: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-1/wal/wals/c09a8aa14cbf4e37b0cd60c52e18c8a8
I20260928 08:53:15.342300 25863 ts_tablet_manager.cc:1939] T c09a8aa14cbf4e37b0cd60c52e18c8a8 P c05be0432c014c318a17e8d389453b91: Deleting consensus metadata
I20260928 08:53:15.342620 25321 catalog_manager.cc:4841] TS c05be0432c014c318a17e8d389453b91 (127.23.167.194:36833): tablet c09a8aa14cbf4e37b0cd60c52e18c8a8 (table test-table [id=4e5ea306145c441f82c41292bad2c5fb]) successfully deleted
W20260928 08:53:15.413910 25539 tablet.cc:2369] T 0d5ceed395c54ecb84e8c236d239e361 P 7417efc3585546f9a1346f5e37609996: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:53:15.456925 25819 tablet.cc:2369] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:53:15.532528 25856 raft_consensus.cc:482] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:53:15.532641 25856 raft_consensus.cc:504] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } }
I20260928 08:53:15.532837 25856 leader_election.cc:288] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7417efc3585546f9a1346f5e37609996 (127.23.167.193:35251), c05be0432c014c318a17e8d389453b91 (127.23.167.194:36833)
I20260928 08:53:15.533120 25492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5ceed395c54ecb84e8c236d239e361" candidate_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7417efc3585546f9a1346f5e37609996" is_pre_election: true
I20260928 08:53:15.533118 25632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5ceed395c54ecb84e8c236d239e361" candidate_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05be0432c014c318a17e8d389453b91" is_pre_election: true
I20260928 08:53:15.533236 25632 raft_consensus.cc:2454] T 0d5ceed395c54ecb84e8c236d239e361 P c05be0432c014c318a17e8d389453b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99fcfc80d4e34d86b21160e36ba1a88b in term 0.
I20260928 08:53:15.533236 25492 raft_consensus.cc:2454] T 0d5ceed395c54ecb84e8c236d239e361 P 7417efc3585546f9a1346f5e37609996 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99fcfc80d4e34d86b21160e36ba1a88b in term 0.
I20260928 08:53:15.533413 25705 leader_election.cc:302] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99fcfc80d4e34d86b21160e36ba1a88b, c05be0432c014c318a17e8d389453b91; no voters:
I20260928 08:53:15.533528 25856 raft_consensus.cc:2781] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:53:15.533586 25856 raft_consensus.cc:482] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:53:15.533600 25856 raft_consensus.cc:3037] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:15.534351 25856 raft_consensus.cc:504] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } }
I20260928 08:53:15.534475 25856 leader_election.cc:288] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b [CANDIDATE]: Term 1 election: Requested vote from peers 7417efc3585546f9a1346f5e37609996 (127.23.167.193:35251), c05be0432c014c318a17e8d389453b91 (127.23.167.194:36833)
I20260928 08:53:15.534663 25632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5ceed395c54ecb84e8c236d239e361" candidate_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05be0432c014c318a17e8d389453b91"
I20260928 08:53:15.534693 25492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5ceed395c54ecb84e8c236d239e361" candidate_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7417efc3585546f9a1346f5e37609996"
I20260928 08:53:15.534754 25632 raft_consensus.cc:3037] T 0d5ceed395c54ecb84e8c236d239e361 P c05be0432c014c318a17e8d389453b91 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:15.534775 25492 raft_consensus.cc:3037] T 0d5ceed395c54ecb84e8c236d239e361 P 7417efc3585546f9a1346f5e37609996 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:15.535532 25632 raft_consensus.cc:2454] T 0d5ceed395c54ecb84e8c236d239e361 P c05be0432c014c318a17e8d389453b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99fcfc80d4e34d86b21160e36ba1a88b in term 1.
I20260928 08:53:15.535532 25492 raft_consensus.cc:2454] T 0d5ceed395c54ecb84e8c236d239e361 P 7417efc3585546f9a1346f5e37609996 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99fcfc80d4e34d86b21160e36ba1a88b in term 1.
I20260928 08:53:15.535714 25706 leader_election.cc:302] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7417efc3585546f9a1346f5e37609996, 99fcfc80d4e34d86b21160e36ba1a88b; no voters:
I20260928 08:53:15.535831 25856 raft_consensus.cc:2781] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:53:15.535902 25856 raft_consensus.cc:686] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b [term 1 LEADER]: Becoming Leader. State: Replica: 99fcfc80d4e34d86b21160e36ba1a88b, State: Running, Role: LEADER
I20260928 08:53:15.535949 25856 consensus_queue.cc:237] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } }
I20260928 08:53:15.536552 25335 catalog_manager.cc:5495] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99fcfc80d4e34d86b21160e36ba1a88b (127.23.167.195). New cstate: current_term: 1 leader_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } health_report { overall_health: HEALTHY } } }
Sep 28 08:53:15 dist-test-slave-pbjp krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585594, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:15.592113 25873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.585723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:43303 (local address 127.0.0.1:36826)
0928 08:53:15.585873 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:15.585884 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:15.585953 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:15.586077 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:15.586077 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:15.586401 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:15.586449 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:15.586451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.587089 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.587090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:15.587646 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:15.587648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.587975 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.587976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:15.588036 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.588499 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:15.588503 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:15.589846 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:15.591242 (+ 1396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:15.591249 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:15.591251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:15.591469 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:15.591612 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:15.591613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:15.591613 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:15.591662 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:15.591831 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:15.591834 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:15.591875 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 08:53:15.591920 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 08:53:15.592019 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":2,"thread_start_us":78,"threads_started":1}
I20260928 08:53:15.592154 25385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.585783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36826 (local address 127.23.167.254:43303)
0928 08:53:15.585845 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:15.585847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:15.585944 (+ 97us) server_negotiation.cc:407] Connection header received
0928 08:53:15.586010 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:15.586011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:15.586021 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:15.586067 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:15.586530 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.587019 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.587725 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.587914 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.587968 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.589962 (+ 1994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:15.589963 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:15.589963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:15.589969 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:15.591170 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:15.591524 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:15.591525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:15.591526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:15.591569 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:15.591700 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:15.591700 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:15.591700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:15.591790 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:15.591836 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:15.591937 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 08:53:15.592058 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":38}
I20260928 08:53:15.594080 25335 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:36826:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 08:53:15.594260 25335 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:53:15.595625 25752 tablet_service.cc:1467] Processing CreateTablet for tablet e6218c8a5db04e24bf80996169b380f1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=89209ee5f39c4164935c022a8811ab31]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:53:15.595636 25472 tablet_service.cc:1467] Processing CreateTablet for tablet e6218c8a5db04e24bf80996169b380f1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=89209ee5f39c4164935c022a8811ab31]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:53:15.595693 25612 tablet_service.cc:1467] Processing CreateTablet for tablet e6218c8a5db04e24bf80996169b380f1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=89209ee5f39c4164935c022a8811ab31]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:53:15.595784 25472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6218c8a5db04e24bf80996169b380f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:15.595785 25752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6218c8a5db04e24bf80996169b380f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:15.595837 25612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6218c8a5db04e24bf80996169b380f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:15.596951 25839 tablet_bootstrap.cc:492] T e6218c8a5db04e24bf80996169b380f1 P c05be0432c014c318a17e8d389453b91: Bootstrap starting.
I20260928 08:53:15.597220 25838 tablet_bootstrap.cc:492] T e6218c8a5db04e24bf80996169b380f1 P 99fcfc80d4e34d86b21160e36ba1a88b: Bootstrap starting.
I20260928 08:53:15.597222 25841 tablet_bootstrap.cc:492] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996: Bootstrap starting.
I20260928 08:53:15.597369 25839 tablet_bootstrap.cc:654] T e6218c8a5db04e24bf80996169b380f1 P c05be0432c014c318a17e8d389453b91: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:15.597504 25838 tablet_bootstrap.cc:654] T e6218c8a5db04e24bf80996169b380f1 P 99fcfc80d4e34d86b21160e36ba1a88b: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:15.597615 25841 tablet_bootstrap.cc:654] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:15.597997 25838 tablet_bootstrap.cc:492] T e6218c8a5db04e24bf80996169b380f1 P 99fcfc80d4e34d86b21160e36ba1a88b: No bootstrap required, opened a new log
I20260928 08:53:15.598052 25838 ts_tablet_manager.cc:1397] T e6218c8a5db04e24bf80996169b380f1 P 99fcfc80d4e34d86b21160e36ba1a88b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 08:53:15.598176 25839 tablet_bootstrap.cc:492] T e6218c8a5db04e24bf80996169b380f1 P c05be0432c014c318a17e8d389453b91: No bootstrap required, opened a new log
I20260928 08:53:15.598172 25838 raft_consensus.cc:348] T e6218c8a5db04e24bf80996169b380f1 P 99fcfc80d4e34d86b21160e36ba1a88b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } } peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } }
I20260928 08:53:15.598217 25839 ts_tablet_manager.cc:1397] T e6218c8a5db04e24bf80996169b380f1 P c05be0432c014c318a17e8d389453b91: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 08:53:15.598232 25838 raft_consensus.cc:374] T e6218c8a5db04e24bf80996169b380f1 P 99fcfc80d4e34d86b21160e36ba1a88b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:15.598248 25838 raft_consensus.cc:729] T e6218c8a5db04e24bf80996169b380f1 P 99fcfc80d4e34d86b21160e36ba1a88b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99fcfc80d4e34d86b21160e36ba1a88b, State: Initialized, Role: FOLLOWER
I20260928 08:53:15.598294 25838 consensus_queue.cc:260] T e6218c8a5db04e24bf80996169b380f1 P 99fcfc80d4e34d86b21160e36ba1a88b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } } peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } }
I20260928 08:53:15.598404 25838 ts_tablet_manager.cc:1428] T e6218c8a5db04e24bf80996169b380f1 P 99fcfc80d4e34d86b21160e36ba1a88b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:15.598371 25839 raft_consensus.cc:348] T e6218c8a5db04e24bf80996169b380f1 P c05be0432c014c318a17e8d389453b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } } peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } }
I20260928 08:53:15.598444 25839 raft_consensus.cc:374] T e6218c8a5db04e24bf80996169b380f1 P c05be0432c014c318a17e8d389453b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:15.598462 25839 raft_consensus.cc:729] T e6218c8a5db04e24bf80996169b380f1 P c05be0432c014c318a17e8d389453b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05be0432c014c318a17e8d389453b91, State: Initialized, Role: FOLLOWER
I20260928 08:53:15.598405 25856 tablet_replica.cc:439] T e6218c8a5db04e24bf80996169b380f1 P 99fcfc80d4e34d86b21160e36ba1a88b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } } peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } } }
I20260928 08:53:15.598500 25839 consensus_queue.cc:260] T e6218c8a5db04e24bf80996169b380f1 P c05be0432c014c318a17e8d389453b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } } peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } }
I20260928 08:53:15.598534 25856 tablet_replica.cc:442] T e6218c8a5db04e24bf80996169b380f1 P 99fcfc80d4e34d86b21160e36ba1a88b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:53:15.598587 25839 ts_tablet_manager.cc:1428] T e6218c8a5db04e24bf80996169b380f1 P c05be0432c014c318a17e8d389453b91: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:15.598639 25845 tablet_replica.cc:439] T e6218c8a5db04e24bf80996169b380f1 P c05be0432c014c318a17e8d389453b91: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } } peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } } }
I20260928 08:53:15.598711 25845 tablet_replica.cc:442] T e6218c8a5db04e24bf80996169b380f1 P c05be0432c014c318a17e8d389453b91: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:53:15.598804 25841 tablet_bootstrap.cc:492] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996: No bootstrap required, opened a new log
I20260928 08:53:15.598845 25841 ts_tablet_manager.cc:1397] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:53:15.598985 25841 raft_consensus.cc:348] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } } peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } }
I20260928 08:53:15.599049 25841 raft_consensus.cc:374] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:15.599067 25841 raft_consensus.cc:729] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7417efc3585546f9a1346f5e37609996, State: Initialized, Role: FOLLOWER
I20260928 08:53:15.599102 25841 consensus_queue.cc:260] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } } peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } }
I20260928 08:53:15.599179 25841 ts_tablet_manager.cc:1428] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:15.599229 25846 tablet_replica.cc:439] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } } peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } } }
I20260928 08:53:15.599347 25846 tablet_replica.cc:442] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:53:15.605538 25846 raft_consensus.cc:482] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:53:15.605610 25846 raft_consensus.cc:504] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } } peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } }
I20260928 08:53:15.605899 25846 leader_election.cc:288] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99fcfc80d4e34d86b21160e36ba1a88b (127.23.167.195:44001), c05be0432c014c318a17e8d389453b91 (127.23.167.194:36833)
I20260928 08:53:15.608723 25837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.606071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:51445 (local address 127.23.167.195:44001)
0928 08:53:15.606124 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:15.606126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:15.606270 (+ 144us) server_negotiation.cc:407] Connection header received
0928 08:53:15.606337 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:15.606338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:15.606350 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:15.606389 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:53:15.606537 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.607020 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.608022 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.608380 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.608438 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.608446 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:15.608618 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 08:53:15.608656 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 08:53:15.608726 25834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.606136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:41521 (local address 127.23.167.194:36833)
0928 08:53:15.606184 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:15.606186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:15.606304 (+ 118us) server_negotiation.cc:407] Connection header received
0928 08:53:15.606368 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:15.606368 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:15.606376 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:15.606419 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:53:15.606633 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.607111 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.608051 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.608404 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.608472 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.608481 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:15.608624 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 08:53:15.608658 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 08:53:15.608779 25878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.606028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:44001 (local address 127.23.167.193:51445)
0928 08:53:15.606202 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:15.606212 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:15.606270 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:15.606409 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:15.606410 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:15.606412 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:53:15.606493 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:15.606494 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.607086 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.607087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:15.607940 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:15.607942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.608486 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.608488 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:15.608550 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.608553 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:53:15.608599 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 08:53:15.608641 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":18,"thread_start_us":81,"threads_started":1}
I20260928 08:53:15.608981 25772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6218c8a5db04e24bf80996169b380f1" candidate_uuid: "7417efc3585546f9a1346f5e37609996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" is_pre_election: true
I20260928 08:53:15.609136 25772 raft_consensus.cc:2454] T e6218c8a5db04e24bf80996169b380f1 P 99fcfc80d4e34d86b21160e36ba1a88b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7417efc3585546f9a1346f5e37609996 in term 0.
I20260928 08:53:15.609149 25877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.606049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:36833 (local address 127.23.167.193:41521)
0928 08:53:15.606251 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:15.606260 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:15.606318 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:15.606462 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:15.606462 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:15.606465 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:53:15.606561 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:15.606563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.607186 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.607187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:15.607970 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:15.607972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.608498 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.608499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:15.608555 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.608558 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:53:15.608602 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 08:53:15.608644 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":18,"thread_start_us":65,"threads_started":1}
I20260928 08:53:15.609416 25632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6218c8a5db04e24bf80996169b380f1" candidate_uuid: "7417efc3585546f9a1346f5e37609996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05be0432c014c318a17e8d389453b91" is_pre_election: true
I20260928 08:53:15.609423 25426 leader_election.cc:302] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7417efc3585546f9a1346f5e37609996, 99fcfc80d4e34d86b21160e36ba1a88b; no voters:
I20260928 08:53:15.609489 25632 raft_consensus.cc:2454] T e6218c8a5db04e24bf80996169b380f1 P c05be0432c014c318a17e8d389453b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7417efc3585546f9a1346f5e37609996 in term 0.
I20260928 08:53:15.609568 25846 raft_consensus.cc:2781] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:53:15.609612 25846 raft_consensus.cc:482] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:53:15.609639 25846 raft_consensus.cc:3037] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:15.610409 25846 raft_consensus.cc:504] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } } peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } }
I20260928 08:53:15.610540 25846 leader_election.cc:288] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996 [CANDIDATE]: Term 1 election: Requested vote from peers 99fcfc80d4e34d86b21160e36ba1a88b (127.23.167.195:44001), c05be0432c014c318a17e8d389453b91 (127.23.167.194:36833)
I20260928 08:53:15.610679 25632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6218c8a5db04e24bf80996169b380f1" candidate_uuid: "7417efc3585546f9a1346f5e37609996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05be0432c014c318a17e8d389453b91"
I20260928 08:53:15.610751 25632 raft_consensus.cc:3037] T e6218c8a5db04e24bf80996169b380f1 P c05be0432c014c318a17e8d389453b91 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:15.611334 25632 raft_consensus.cc:2454] T e6218c8a5db04e24bf80996169b380f1 P c05be0432c014c318a17e8d389453b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7417efc3585546f9a1346f5e37609996 in term 1.
I20260928 08:53:15.611446 25772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6218c8a5db04e24bf80996169b380f1" candidate_uuid: "7417efc3585546f9a1346f5e37609996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99fcfc80d4e34d86b21160e36ba1a88b"
I20260928 08:53:15.611524 25772 raft_consensus.cc:3037] T e6218c8a5db04e24bf80996169b380f1 P 99fcfc80d4e34d86b21160e36ba1a88b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:15.611516 25425 leader_election.cc:302] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7417efc3585546f9a1346f5e37609996, c05be0432c014c318a17e8d389453b91; no voters:
I20260928 08:53:15.611598 25846 raft_consensus.cc:2781] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:53:15.611732 25846 raft_consensus.cc:686] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996 [term 1 LEADER]: Becoming Leader. State: Replica: 7417efc3585546f9a1346f5e37609996, State: Running, Role: LEADER
I20260928 08:53:15.611819 25846 consensus_queue.cc:237] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } } peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } }
I20260928 08:53:15.612182 25879 tablet_replica.cc:439] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996: TxnStatusTablet state changed. Reason: New leader 7417efc3585546f9a1346f5e37609996. Latest consensus state: current_term: 1 leader_uuid: "7417efc3585546f9a1346f5e37609996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } } peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } } }
I20260928 08:53:15.612238 25772 raft_consensus.cc:2454] T e6218c8a5db04e24bf80996169b380f1 P 99fcfc80d4e34d86b21160e36ba1a88b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7417efc3585546f9a1346f5e37609996 in term 1.
I20260928 08:53:15.612335 25879 tablet_replica.cc:442] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:53:15.612571 25881 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 08:53:15.612514 25334 catalog_manager.cc:5495] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7417efc3585546f9a1346f5e37609996 (127.23.167.193). New cstate: current_term: 1 leader_uuid: "7417efc3585546f9a1346f5e37609996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } health_report { overall_health: UNKNOWN } } }
I20260928 08:53:15.612675 25881 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 08:53:15.618600 25873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.615700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:35251 (local address 127.0.0.1:37310)
0928 08:53:15.615772 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:15.615780 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:15.615814 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:15.615969 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:15.615970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:15.615971 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:53:15.616034 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:15.616036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.616935 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.616936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:15.617596 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:15.617599 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.618066 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.618068 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:15.618126 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.618142 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:53:15.618435 (+ 293us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:53:15.618447 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 08:53:15.618493 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 08:53:15.618542 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 08:53:15.618602 25835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.615769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37310 (local address 127.23.167.193:35251)
0928 08:53:15.615825 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:15.615827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:15.615837 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:53:15.615868 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:15.615869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:15.615887 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:15.615970 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:53:15.616083 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.616838 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.617671 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.617975 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.618049 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.618231 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:53:15.618353 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:53:15.618431 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:15.618506 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 08:53:15.618528 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 08:53:15.619635 25632 raft_consensus.cc:1261] T e6218c8a5db04e24bf80996169b380f1 P c05be0432c014c318a17e8d389453b91 [term 1 FOLLOWER]: Refusing update from remote peer 7417efc3585546f9a1346f5e37609996: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:53:15.619740 25772 raft_consensus.cc:1261] T e6218c8a5db04e24bf80996169b380f1 P 99fcfc80d4e34d86b21160e36ba1a88b [term 1 FOLLOWER]: Refusing update from remote peer 7417efc3585546f9a1346f5e37609996: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:53:15.619998 25846 consensus_queue.cc:1035] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:53:15.620132 25879 consensus_queue.cc:1035] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:53:15.620668 25845 tablet_replica.cc:439] T e6218c8a5db04e24bf80996169b380f1 P c05be0432c014c318a17e8d389453b91: TxnStatusTablet state changed. Reason: New leader 7417efc3585546f9a1346f5e37609996. Latest consensus state: current_term: 1 leader_uuid: "7417efc3585546f9a1346f5e37609996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } } peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } } }
I20260928 08:53:15.620767 25845 tablet_replica.cc:442] T e6218c8a5db04e24bf80996169b380f1 P c05be0432c014c318a17e8d389453b91: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:53:15.620828 25856 tablet_replica.cc:439] T e6218c8a5db04e24bf80996169b380f1 P 99fcfc80d4e34d86b21160e36ba1a88b: TxnStatusTablet state changed. Reason: New leader 7417efc3585546f9a1346f5e37609996. Latest consensus state: current_term: 1 leader_uuid: "7417efc3585546f9a1346f5e37609996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } } peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } } }
I20260928 08:53:15.620918 25856 tablet_replica.cc:442] T e6218c8a5db04e24bf80996169b380f1 P 99fcfc80d4e34d86b21160e36ba1a88b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:53:15.621842 25846 tablet_replica.cc:439] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7417efc3585546f9a1346f5e37609996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } } peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } } }
I20260928 08:53:15.621948 25846 tablet_replica.cc:442] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:53:15.621850 25886 tablet_replica.cc:439] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7417efc3585546f9a1346f5e37609996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } } peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } } }
I20260928 08:53:15.622102 25886 tablet_replica.cc:442] T e6218c8a5db04e24bf80996169b380f1 P 7417efc3585546f9a1346f5e37609996: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:53:15.622077 25856 tablet_replica.cc:439] T e6218c8a5db04e24bf80996169b380f1 P 99fcfc80d4e34d86b21160e36ba1a88b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7417efc3585546f9a1346f5e37609996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } } peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } } }
I20260928 08:53:15.622159 25856 tablet_replica.cc:442] T e6218c8a5db04e24bf80996169b380f1 P 99fcfc80d4e34d86b21160e36ba1a88b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:53:15.622615 25852 mvcc.cc:205] Tried to move back new op lower bound from 7334238599656079360 to 7334238599626555392. Current Snapshot: MvccSnapshot[applied={T|T < 7334238599656079360}]
I20260928 08:53:15.623140 25845 tablet_replica.cc:439] T e6218c8a5db04e24bf80996169b380f1 P c05be0432c014c318a17e8d389453b91: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7417efc3585546f9a1346f5e37609996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 } } peers { permanent_uuid: "99fcfc80d4e34d86b21160e36ba1a88b" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44001 } } peers { permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 } } }
I20260928 08:53:15.623229 25845 tablet_replica.cc:442] T e6218c8a5db04e24bf80996169b380f1 P c05be0432c014c318a17e8d389453b91: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:53:15.628124 25835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.625724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:59099 (local address 127.23.167.193:35251)
0928 08:53:15.625762 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:15.625764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:15.625770 (+ 6us) server_negotiation.cc:407] Connection header received
0928 08:53:15.625787 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:15.625787 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:15.625796 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:15.625841 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:53:15.625974 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.626428 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.627534 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.627715 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.627773 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.627931 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:53:15.627957 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:53:15.627989 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:15.628049 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 08:53:15.628065 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 08:53:15.628175 25824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.625541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:35251 (local address 127.23.167.195:59099)
0928 08:53:15.625613 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:15.625623 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:15.625657 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:15.625864 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:15.625865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:15.625867 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:53:15.625917 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:15.625919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.626496 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.626496 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:15.627447 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:15.627449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.627780 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.627781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:15.627846 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.627866 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:53:15.628000 (+ 134us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:53:15.628001 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 08:53:15.628046 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 08:53:15.628095 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 08:53:15.630244 25632 raft_consensus.cc:1261] T 0d5ceed395c54ecb84e8c236d239e361 P c05be0432c014c318a17e8d389453b91 [term 1 FOLLOWER]: Refusing update from remote peer 99fcfc80d4e34d86b21160e36ba1a88b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:53:15.630268 25492 raft_consensus.cc:1261] T 0d5ceed395c54ecb84e8c236d239e361 P 7417efc3585546f9a1346f5e37609996 [term 1 FOLLOWER]: Refusing update from remote peer 99fcfc80d4e34d86b21160e36ba1a88b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:53:15.630434 25856 consensus_queue.cc:1035] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05be0432c014c318a17e8d389453b91" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:53:15.630522 25848 consensus_queue.cc:1035] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7417efc3585546f9a1346f5e37609996" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:53:15.638624 25837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.635849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:34207 (local address 127.23.167.195:44001)
0928 08:53:15.635900 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:15.635902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:15.635951 (+ 49us) server_negotiation.cc:407] Connection header received
0928 08:53:15.636012 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:15.636013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:15.636024 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:15.636075 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:53:15.636315 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.636776 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.637928 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.638205 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.638259 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.638425 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:53:15.638452 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:53:15.638485 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:15.638548 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 08:53:15.638567 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 08:53:15.638695 25892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.635713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:44001 (local address 127.23.167.193:34207)
0928 08:53:15.635890 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:15.635902 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:15.635965 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:15.636136 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:15.636137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:15.636139 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:53:15.636256 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:15.636257 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.636846 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.636846 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:15.637851 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:15.637854 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.638267 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.638268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:15.638334 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.638353 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:53:15.638497 (+ 144us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:53:15.638499 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 08:53:15.638545 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 08:53:15.638591 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":89,"threads_started":1}
I20260928 08:53:15.647285 25333 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:53:15.647428 25333 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:53:15.648075 25333 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c3f8ca93a784f60866c3b4c167eda0a: Sending DeleteTablet for 3 replicas of tablet 0d5ceed395c54ecb84e8c236d239e361
I20260928 08:53:15.648375 25752 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d5ceed395c54ecb84e8c236d239e361 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:53:15 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:47434
I20260928 08:53:15.648388 25472 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d5ceed395c54ecb84e8c236d239e361 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:53:15 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:37306
I20260928 08:53:15.648480 25861 tablet_replica.cc:330] T 0d5ceed395c54ecb84e8c236d239e361 P 7417efc3585546f9a1346f5e37609996: stopping tablet replica
I20260928 08:53:15.648480 25862 tablet_replica.cc:330] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b: stopping tablet replica
I20260928 08:53:15.648527 25861 raft_consensus.cc:2229] T 0d5ceed395c54ecb84e8c236d239e361 P 7417efc3585546f9a1346f5e37609996 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:53:15.648535 25862 raft_consensus.cc:2229] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:53:15.648561 25861 raft_consensus.cc:2258] T 0d5ceed395c54ecb84e8c236d239e361 P 7417efc3585546f9a1346f5e37609996 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:53:15.648591 25862 raft_consensus.cc:2258] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:53:15.648783 25612 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d5ceed395c54ecb84e8c236d239e361 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:53:15 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:54590
I20260928 08:53:15.648896 25863 tablet_replica.cc:330] T 0d5ceed395c54ecb84e8c236d239e361 P c05be0432c014c318a17e8d389453b91: stopping tablet replica
I20260928 08:53:15.648947 25863 raft_consensus.cc:2229] T 0d5ceed395c54ecb84e8c236d239e361 P c05be0432c014c318a17e8d389453b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:53:15.648988 25863 raft_consensus.cc:2258] T 0d5ceed395c54ecb84e8c236d239e361 P c05be0432c014c318a17e8d389453b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:53:15.648979 25861 ts_tablet_manager.cc:1905] T 0d5ceed395c54ecb84e8c236d239e361 P 7417efc3585546f9a1346f5e37609996: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:53:15.649317 25863 ts_tablet_manager.cc:1905] T 0d5ceed395c54ecb84e8c236d239e361 P c05be0432c014c318a17e8d389453b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:53:15.649353 25862 ts_tablet_manager.cc:1905] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:53:15.651352 25863 ts_tablet_manager.cc:1918] T 0d5ceed395c54ecb84e8c236d239e361 P c05be0432c014c318a17e8d389453b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:53:15.651414 25863 log.cc:1196] T 0d5ceed395c54ecb84e8c236d239e361 P c05be0432c014c318a17e8d389453b91: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-1/wal/wals/0d5ceed395c54ecb84e8c236d239e361
I20260928 08:53:15.651607 25863 ts_tablet_manager.cc:1939] T 0d5ceed395c54ecb84e8c236d239e361 P c05be0432c014c318a17e8d389453b91: Deleting consensus metadata
I20260928 08:53:15.652009 25321 catalog_manager.cc:4841] TS c05be0432c014c318a17e8d389453b91 (127.23.167.194:36833): tablet 0d5ceed395c54ecb84e8c236d239e361 (table test-table [id=620942273acc4683b832164bf6b36ecf]) successfully deleted
I20260928 08:53:15.652561 25862 ts_tablet_manager.cc:1918] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:53:15.652627 25862 log.cc:1196] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-2/wal/wals/0d5ceed395c54ecb84e8c236d239e361
I20260928 08:53:15.652848 25862 ts_tablet_manager.cc:1939] T 0d5ceed395c54ecb84e8c236d239e361 P 99fcfc80d4e34d86b21160e36ba1a88b: Deleting consensus metadata
I20260928 08:53:15.653174 25861 ts_tablet_manager.cc:1918] T 0d5ceed395c54ecb84e8c236d239e361 P 7417efc3585546f9a1346f5e37609996: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:53:15.653174 25322 catalog_manager.cc:4841] TS 99fcfc80d4e34d86b21160e36ba1a88b (127.23.167.195:44001): tablet 0d5ceed395c54ecb84e8c236d239e361 (table test-table [id=620942273acc4683b832164bf6b36ecf]) successfully deleted
I20260928 08:53:15.653228 25861 log.cc:1196] T 0d5ceed395c54ecb84e8c236d239e361 P 7417efc3585546f9a1346f5e37609996: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585589999956-24223-0/minicluster-data/ts-0/wal/wals/0d5ceed395c54ecb84e8c236d239e361
I20260928 08:53:15.653424 25861 ts_tablet_manager.cc:1939] T 0d5ceed395c54ecb84e8c236d239e361 P 7417efc3585546f9a1346f5e37609996: Deleting consensus metadata
I20260928 08:53:15.653717 25322 catalog_manager.cc:4841] TS 7417efc3585546f9a1346f5e37609996 (127.23.167.193:35251): tablet 0d5ceed395c54ecb84e8c236d239e361 (table test-table [id=620942273acc4683b832164bf6b36ecf]) successfully deleted
Sep 28 08:53:15 dist-test-slave-pbjp krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585595, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.167.193@KRBTEST.COM
I20260928 08:53:15.659451 25835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.650960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37314 (local address 127.23.167.193:35251)
0928 08:53:15.651004 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:15.651006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:15.651173 (+ 167us) server_negotiation.cc:407] Connection header received
0928 08:53:15.651233 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:15.651233 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:15.651244 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:15.651288 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:15.651983 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.652507 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.653416 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.653678 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.653720 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.656845 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:15.656852 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:15.656853 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:15.656859 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:15.658394 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:15.658772 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:15.658774 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:15.658774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:15.658826 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:15.658992 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:15.658992 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:15.658992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:15.659117 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:15.659229 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:15.659314 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 08:53:15.659373 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 08:53:15.659821 25532 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37314
I20260928 08:53:15.667009 25385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:15.660709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36834 (local address 127.23.167.254:43303)
0928 08:53:15.660753 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:15.660755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:15.660782 (+ 27us) server_negotiation.cc:407] Connection header received
0928 08:53:15.660836 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:15.660837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:15.660847 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:15.660906 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:15.661384 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.662159 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.662854 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:15.663162 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:15.663230 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:15.664287 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:15.664288 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:15.664289 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:15.664296 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:15.665900 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:15.666193 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:15.666194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:15.666194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:15.666253 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:15.666415 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:15.666415 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:15.666416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:15.666543 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:15.666638 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:15.666777 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 08:53:15.666908 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 08:53:15.667215 25333 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36834
I20260928 08:53:15.667922 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 25407
I20260928 08:53:15.673629 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 25547
I20260928 08:53:15.677484 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 25687
I20260928 08:53:15.682387 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 25303
2026-09-28T08:53:15Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2135 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:53:15 dist-test-slave-pbjp krb5kdc[25900](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:53:15 dist-test-slave-pbjp krb5kdc[25900](info): set up 2 sockets
Sep 28 08:53:15 dist-test-slave-pbjp krb5kdc[25900](info): commencing operation
krb5kdc: starting...
W20260928 08:53:16.632591 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:53:16 dist-test-slave-pbjp krb5kdc[25900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:53:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:53:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.167.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
I20260928 08:53:16.744248 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:46595
--webserver_interface=127.23.167.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:35387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:46595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:16.824007 25913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:16.824209 25913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:16.824230 25913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:16.825349 25913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:53:16.825384 25913 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 08:53:16.825395 25913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:16.825407 25913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:16.825417 25913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:53:16.825443 25913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:53:16.826839 25913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:35387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:46595
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:46595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:16.827037 25913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:16.827266 25913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:16.831807 25921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:16.831802 25918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:16.831964 25913 server_base.cc:1034] running on GCE node
W20260928 08:53:16.831818 25919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:16.832235 25913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:16.832516 25913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:16.833647 25913 hybrid_clock.cc:648] HybridClock initialized: now 1790585596833641 us; error 30 us; skew 500 ppm
Sep 28 08:53:16 dist-test-slave-pbjp krb5kdc[25900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585596, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:16.835934 25913 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:53:16.836345 25913 webserver.cc:460] Webserver started at http://127.23.167.254:44627/ using document root <none> and password file <none>
I20260928 08:53:16.836560 25913 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:16.836607 25913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:16.836728 25913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:16.837795 25913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "e718379167694065b3c67180d47b82c0"
format_stamp: "Formatted at 2026-09-28 08:53:16 on dist-test-slave-pbjp"
server_key: "045c85cc37ee79e2ca28903ad2d75894"
server_key_iv: "ffb6025b8d6f03819d4f7c82188ddea8"
server_key_version: "encryptionkey@0"
I20260928 08:53:16.838128 25913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "e718379167694065b3c67180d47b82c0"
format_stamp: "Formatted at 2026-09-28 08:53:16 on dist-test-slave-pbjp"
server_key: "045c85cc37ee79e2ca28903ad2d75894"
server_key_iv: "ffb6025b8d6f03819d4f7c82188ddea8"
server_key_version: "encryptionkey@0"
I20260928 08:53:16.839391 25913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:53:16.840055 25928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:16.840221 25913 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:53:16.840302 25913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "e718379167694065b3c67180d47b82c0"
format_stamp: "Formatted at 2026-09-28 08:53:16 on dist-test-slave-pbjp"
server_key: "045c85cc37ee79e2ca28903ad2d75894"
server_key_iv: "ffb6025b8d6f03819d4f7c82188ddea8"
server_key_version: "encryptionkey@0"
I20260928 08:53:16.840364 25913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:16.880717 25913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:16.883018 25913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:16.883163 25913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:16.887557 25913 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:46595
I20260928 08:53:16.887640 25990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:46595 every 8 connection(s)
I20260928 08:53:16.887940 25913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:53:16.888581 25991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:16.889175 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 25913
I20260928 08:53:16.889256 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:53:16.889426 24223 external_mini_cluster.cc:1428] Setting key 2e76afe61dc453c8e002ba10f8fd72be
I20260928 08:53:16.891490 25991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e718379167694065b3c67180d47b82c0: Bootstrap starting.
I20260928 08:53:16.892067 25991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e718379167694065b3c67180d47b82c0: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:16.892331 25991 log.cc:824] T 00000000000000000000000000000000 P e718379167694065b3c67180d47b82c0: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:16.893015 25991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e718379167694065b3c67180d47b82c0: No bootstrap required, opened a new log
I20260928 08:53:16.895275 25991 raft_consensus.cc:348] T 00000000000000000000000000000000 P e718379167694065b3c67180d47b82c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e718379167694065b3c67180d47b82c0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46595 } }
I20260928 08:53:16.895453 25991 raft_consensus.cc:374] T 00000000000000000000000000000000 P e718379167694065b3c67180d47b82c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:16.895491 25991 raft_consensus.cc:729] T 00000000000000000000000000000000 P e718379167694065b3c67180d47b82c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e718379167694065b3c67180d47b82c0, State: Initialized, Role: FOLLOWER
I20260928 08:53:16.895638 25991 consensus_queue.cc:260] T 00000000000000000000000000000000 P e718379167694065b3c67180d47b82c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e718379167694065b3c67180d47b82c0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46595 } }
I20260928 08:53:16.895745 25991 raft_consensus.cc:388] T 00000000000000000000000000000000 P e718379167694065b3c67180d47b82c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:53:16.895777 25991 raft_consensus.cc:482] T 00000000000000000000000000000000 P e718379167694065b3c67180d47b82c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:53:16.895816 25991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e718379167694065b3c67180d47b82c0 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 08:53:16 dist-test-slave-pbjp krb5kdc[25900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:16.896401 25991 raft_consensus.cc:504] T 00000000000000000000000000000000 P e718379167694065b3c67180d47b82c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e718379167694065b3c67180d47b82c0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46595 } }
I20260928 08:53:16.896508 25991 leader_election.cc:302] T 00000000000000000000000000000000 P e718379167694065b3c67180d47b82c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e718379167694065b3c67180d47b82c0; no voters:
I20260928 08:53:16.896677 25991 leader_election.cc:288] T 00000000000000000000000000000000 P e718379167694065b3c67180d47b82c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:53:16.896719 25996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e718379167694065b3c67180d47b82c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:53:16.896883 25996 raft_consensus.cc:686] T 00000000000000000000000000000000 P e718379167694065b3c67180d47b82c0 [term 1 LEADER]: Becoming Leader. State: Replica: e718379167694065b3c67180d47b82c0, State: Running, Role: LEADER
I20260928 08:53:16.896956 25991 sys_catalog.cc:564] T 00000000000000000000000000000000 P e718379167694065b3c67180d47b82c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:53:16.896971 25996 consensus_queue.cc:237] T 00000000000000000000000000000000 P e718379167694065b3c67180d47b82c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e718379167694065b3c67180d47b82c0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46595 } }
I20260928 08:53:16.897321 25996 sys_catalog.cc:455] T 00000000000000000000000000000000 P e718379167694065b3c67180d47b82c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e718379167694065b3c67180d47b82c0. Latest consensus state: current_term: 1 leader_uuid: "e718379167694065b3c67180d47b82c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e718379167694065b3c67180d47b82c0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46595 } } }
I20260928 08:53:16.897397 25996 sys_catalog.cc:458] T 00000000000000000000000000000000 P e718379167694065b3c67180d47b82c0 [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:16.897737 26001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:53:16.898239 26001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:53:16.898727 25997 sys_catalog.cc:455] T 00000000000000000000000000000000 P e718379167694065b3c67180d47b82c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e718379167694065b3c67180d47b82c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e718379167694065b3c67180d47b82c0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46595 } } }
I20260928 08:53:16.898820 25997 sys_catalog.cc:458] T 00000000000000000000000000000000 P e718379167694065b3c67180d47b82c0 [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:16.899907 26001 catalog_manager.cc:1347] Generated new cluster ID: b086671f735a435c8a06a905621791fb
I20260928 08:53:16.899957 26001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:53:16.900224 25994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:16.889932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39528 (local address 127.23.167.254:46595)
0928 08:53:16.890084 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:16.890093 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:16.890120 (+ 27us) server_negotiation.cc:407] Connection header received
0928 08:53:16.890526 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:16.890532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:16.890578 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:16.890694 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:16.891416 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:16.892372 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:16.893253 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:16.893881 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:16.894004 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:16.896432 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:16.896437 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:16.896437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:16.896451 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:16.898607 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:16.899033 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:16.899036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:16.899036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:16.899123 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:16.899328 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:16.899329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:16.899329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:16.899630 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:16.899764 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:16.899885 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 08:53:16.900027 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260928 08:53:16.915215 26001 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:53:16.915971 26001 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:53:16.922513 26001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e718379167694065b3c67180d47b82c0: Generated new TSK 0
I20260928 08:53:16.922784 26001 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.167.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
I20260928 08:53:16.953177 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:0
--local_ip_for_outbound_sockets=127.23.167.193
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:46595
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:35387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:17.034066 26017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:17.034269 26017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:17.034292 26017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:17.035427 26017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:17.035466 26017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:17.035485 26017 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 08:53:17.035496 26017 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:53:17.035511 26017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:53:17.036939 26017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:35387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.167.254:46595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:17.037199 26017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:17.037412 26017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:17.042145 26025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:17.042156 26022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:17.042343 26023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:17.042395 26017 server_base.cc:1034] running on GCE node
I20260928 08:53:17.042606 26017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:17.042832 26017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:17.043979 26017 hybrid_clock.cc:648] HybridClock initialized: now 1790585597043962 us; error 30 us; skew 500 ppm
Sep 28 08:53:17 dist-test-slave-pbjp krb5kdc[25900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585597, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:17.046388 26017 init.cc:377] Logged in from keytab as kudu/127.23.167.193@KRBTEST.COM (short username kudu)
I20260928 08:53:17.046849 26017 webserver.cc:460] Webserver started at http://127.23.167.193:35375/ using document root <none> and password file <none>
I20260928 08:53:17.047062 26017 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:17.047107 26017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:17.047236 26017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:17.048261 26017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-0/data/instance:
uuid: "10d5981ef80e4fc5a634c389c7b512f0"
format_stamp: "Formatted at 2026-09-28 08:53:17 on dist-test-slave-pbjp"
server_key: "74efa6d92fb6f5f462c121f5af44deba"
server_key_iv: "0a676bc33bddafe5627d9571cc492b2b"
server_key_version: "encryptionkey@0"
I20260928 08:53:17.048595 26017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance:
uuid: "10d5981ef80e4fc5a634c389c7b512f0"
format_stamp: "Formatted at 2026-09-28 08:53:17 on dist-test-slave-pbjp"
server_key: "74efa6d92fb6f5f462c121f5af44deba"
server_key_iv: "0a676bc33bddafe5627d9571cc492b2b"
server_key_version: "encryptionkey@0"
I20260928 08:53:17.049870 26017 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:53:17.050549 26032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:17.050698 26017 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:53:17.050779 26017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "10d5981ef80e4fc5a634c389c7b512f0"
format_stamp: "Formatted at 2026-09-28 08:53:17 on dist-test-slave-pbjp"
server_key: "74efa6d92fb6f5f462c121f5af44deba"
server_key_iv: "0a676bc33bddafe5627d9571cc492b2b"
server_key_version: "encryptionkey@0"
I20260928 08:53:17.050841 26017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:17.070725 26017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:17.072654 26017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:17.072798 26017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:17.073407 26017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:17.073458 26017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:17.073489 26017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:17.073506 26017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:53:17 dist-test-slave-pbjp krb5kdc[25900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585597, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:17.080258 26017 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:42119
I20260928 08:53:17.080333 26147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:42119 every 8 connection(s)
I20260928 08:53:17.080662 26017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
I20260928 08:53:17.082610 25994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.073824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:37499 (local address 127.23.167.254:46595)
0928 08:53:17.073873 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:17.073876 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:17.074141 (+ 265us) server_negotiation.cc:407] Connection header received
0928 08:53:17.074642 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:17.074644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:17.074657 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:17.074716 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:17.075648 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.076155 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.077640 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.077862 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.077951 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.080086 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:17.080088 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:17.080089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:17.080097 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:17.081544 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:17.081884 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:17.081885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:17.081885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:17.081938 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:17.082093 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:17.082093 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:17.082093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:17.082186 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:17.082228 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:17.082470 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 08:53:17.082544 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 08:53:17.082706 26041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.073823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:46595 (local address 127.23.167.193:37499)
0928 08:53:17.073983 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:17.074021 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:17.074531 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:17.074848 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:17.074853 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:17.075228 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:17.075533 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.075540 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.076226 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.076227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.077562 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.077565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.077976 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.077978 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.078122 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.078609 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:17.078617 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:17.079994 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:17.081611 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:17.081615 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:17.081617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:17.081820 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:17.081978 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:17.081979 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:17.081979 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:17.082060 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:17.082225 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:17.082228 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:17.082356 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 08:53:17.082434 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 08:53:17.082548 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20260928 08:53:17.083143 26148 heartbeater.cc:347] Connected to a master server at 127.23.167.254:46595
I20260928 08:53:17.083225 26148 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:17.083387 26148 heartbeater.cc:511] Master 127.23.167.254:46595 requested a full tablet report, sending...
I20260928 08:53:17.083885 25944 ts_manager.cc:194] Registered new tserver with Master: 10d5981ef80e4fc5a634c389c7b512f0 (127.23.167.193:42119)
I20260928 08:53:17.084728 25944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.193@KRBTEST.COM'} at 127.23.167.193:37499
I20260928 08:53:17.088243 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 26017
I20260928 08:53:17.088339 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance
I20260928 08:53:17.088496 24223 external_mini_cluster.cc:1428] Setting key 5ec58cf3059cdfde48eb0bdf856ef490
I20260928 08:53:17.088992 25994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.084073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:55583 (local address 127.23.167.254:46595)
0928 08:53:17.084112 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:17.084114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:17.084197 (+ 83us) server_negotiation.cc:407] Connection header received
0928 08:53:17.084263 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:17.084263 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:17.084270 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:17.084313 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:17.084613 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.085209 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.085973 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.086166 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.086219 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.086962 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:17.086962 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:17.086963 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:17.086967 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:17.087958 (+ 991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:17.088243 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:17.088244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:17.088245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:17.088336 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:17.088530 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:17.088530 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:17.088531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:17.088653 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:17.088715 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:17.088869 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 08:53:17.088936 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 08:53:17.089049 26154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.084019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:46595 (local address 127.23.167.193:55583)
0928 08:53:17.084142 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:17.084153 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:17.084211 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:17.084318 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:17.084319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:17.084500 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:17.084559 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.084561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.085296 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.085297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.085892 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.085894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.086223 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.086223 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.086274 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.086614 (+ 340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:17.086619 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:17.086898 (+ 279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:17.088018 (+ 1120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:17.088019 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:17.088019 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:17.088167 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:17.088401 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:17.088402 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:17.088403 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:17.088472 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:17.088712 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:17.088713 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:17.088761 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 08:53:17.088849 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 08:53:17.088947 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":1,"thread_start_us":63,"threads_started":1}
WARNING: no policy specified for kudu/127.23.167.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
I20260928 08:53:17.111625 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:0
--local_ip_for_outbound_sockets=127.23.167.194
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:46595
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:35387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:17.191741 26157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:17.191956 26157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:17.191982 26157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:17.193465 26157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:17.193508 26157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:17.193529 26157 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 08:53:17.193539 26157 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:53:17.193555 26157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:53:17.194979 26157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:35387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.167.254:46595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:17.195178 26157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:17.195386 26157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:17.200461 26165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:17.200484 26163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:17.200547 26162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:17.200632 26157 server_base.cc:1034] running on GCE node
I20260928 08:53:17.200815 26157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:17.201052 26157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:17.202227 26157 hybrid_clock.cc:648] HybridClock initialized: now 1790585597202217 us; error 27 us; skew 500 ppm
Sep 28 08:53:17 dist-test-slave-pbjp krb5kdc[25900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585597, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:17.204530 26157 init.cc:377] Logged in from keytab as kudu/127.23.167.194@KRBTEST.COM (short username kudu)
I20260928 08:53:17.204965 26157 webserver.cc:460] Webserver started at http://127.23.167.194:36423/ using document root <none> and password file <none>
I20260928 08:53:17.205190 26157 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:17.205235 26157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:17.205329 26157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:17.206314 26157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-1/data/instance:
uuid: "8891e45017924cd794b7860438bb8cc0"
format_stamp: "Formatted at 2026-09-28 08:53:17 on dist-test-slave-pbjp"
server_key: "347fdd0d9023953456fe475a9e6bb097"
server_key_iv: "ca945c0c68afa207ed6e51366c7e5aec"
server_key_version: "encryptionkey@0"
I20260928 08:53:17.206609 26157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance:
uuid: "8891e45017924cd794b7860438bb8cc0"
format_stamp: "Formatted at 2026-09-28 08:53:17 on dist-test-slave-pbjp"
server_key: "347fdd0d9023953456fe475a9e6bb097"
server_key_iv: "ca945c0c68afa207ed6e51366c7e5aec"
server_key_version: "encryptionkey@0"
I20260928 08:53:17.207760 26157 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 08:53:17.208379 26172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:17.208549 26157 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:53:17.208628 26157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "8891e45017924cd794b7860438bb8cc0"
format_stamp: "Formatted at 2026-09-28 08:53:17 on dist-test-slave-pbjp"
server_key: "347fdd0d9023953456fe475a9e6bb097"
server_key_iv: "ca945c0c68afa207ed6e51366c7e5aec"
server_key_version: "encryptionkey@0"
I20260928 08:53:17.208690 26157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:17.235175 26157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:17.237053 26157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:17.237226 26157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:17.237752 26157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:17.237794 26157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:17.237833 26157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:17.237860 26157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:17.244253 26157 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:33449
I20260928 08:53:17.244315 26287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:33449 every 8 connection(s)
Sep 28 08:53:17 dist-test-slave-pbjp krb5kdc[25900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585597, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:17.244652 26157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
I20260928 08:53:17.246371 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 26157
I20260928 08:53:17.246457 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance
I20260928 08:53:17.246615 24223 external_mini_cluster.cc:1428] Setting key 1e55f727ba09bf1e7cd46d70b4419abd
I20260928 08:53:17.247705 25994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.238132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:46141 (local address 127.23.167.254:46595)
0928 08:53:17.238185 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:17.238187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:17.238408 (+ 221us) server_negotiation.cc:407] Connection header received
0928 08:53:17.238912 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:17.238914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:17.238927 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:17.238966 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:17.239785 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.240567 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.241787 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.242090 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.242168 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.244966 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:17.244967 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:17.244968 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:17.244975 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:17.246278 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:17.246727 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:17.246729 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:17.246729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:17.246794 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:17.247047 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:17.247047 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:17.247048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:17.247195 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:17.247259 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:17.247545 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 08:53:17.247617 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 08:53:17.247795 26181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.238144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:46595 (local address 127.23.167.194:46141)
0928 08:53:17.238294 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:17.238330 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:17.238818 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:17.239051 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:17.239055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:17.239388 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:17.239671 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.239678 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.240668 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.240669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.241713 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.241716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.242192 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.242193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.242331 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.243331 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:17.243342 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:17.244858 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:17.246374 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:17.246380 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:17.246382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:17.246660 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:17.246854 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:17.246854 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:17.246855 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:17.246978 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:17.247260 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:17.247267 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:17.247417 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 08:53:17.247504 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 08:53:17.247624 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
I20260928 08:53:17.248835 26288 heartbeater.cc:347] Connected to a master server at 127.23.167.254:46595
I20260928 08:53:17.248919 26288 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:17.249115 26288 heartbeater.cc:511] Master 127.23.167.254:46595 requested a full tablet report, sending...
I20260928 08:53:17.249437 25945 ts_manager.cc:194] Registered new tserver with Master: 8891e45017924cd794b7860438bb8cc0 (127.23.167.194:33449)
I20260928 08:53:17.249905 25945 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.194@KRBTEST.COM'} at 127.23.167.194:46141
WARNING: no policy specified for kudu/127.23.167.195@KRBTEST.COM; defaulting to no policy
I20260928 08:53:17.255779 26295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.249250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:46595 (local address 127.23.167.194:50289)
0928 08:53:17.250039 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:17.250051 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:17.250110 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:17.250215 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:17.250216 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:17.250399 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:17.250462 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.250463 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.251359 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.251359 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.252155 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.252158 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.252527 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.252528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.252604 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.253014 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:17.253020 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:17.253402 (+ 382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:17.254877 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:17.254878 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:17.254879 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:17.255091 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:17.255263 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:17.255264 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:17.255264 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:17.255316 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:17.255549 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:17.255550 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:17.255587 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 08:53:17.255632 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 08:53:17.255706 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":63,"threads_started":1}
I20260928 08:53:17.255796 25994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.249247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:50289 (local address 127.23.167.254:46595)
0928 08:53:17.249290 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:17.249292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:17.250115 (+ 823us) server_negotiation.cc:407] Connection header received
0928 08:53:17.250147 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:17.250148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:17.250159 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:17.250212 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:17.250545 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.251246 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.252251 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.252449 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.252509 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.253522 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:17.253523 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:17.253524 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:17.253530 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:17.254807 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:17.255158 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:17.255158 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:17.255159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:17.255207 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:17.255355 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:17.255355 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:17.255355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:17.255504 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:17.255551 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:17.255649 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 08:53:17.255707 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
I20260928 08:53:17.269706 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:0
--local_ip_for_outbound_sockets=127.23.167.195
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:46595
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:35387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:17.349380 26297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:17.349581 26297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:17.349614 26297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:17.350750 26297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:17.350791 26297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:17.350819 26297 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 08:53:17.350834 26297 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:53:17.350869 26297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:53:17.352342 26297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:35387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.167.254:46595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:17.352584 26297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:17.352801 26297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:17.357745 26302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:17.357758 26303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:17.357745 26305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:17.357868 26297 server_base.cc:1034] running on GCE node
I20260928 08:53:17.358167 26297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:17.358450 26297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:17.359597 26297 hybrid_clock.cc:648] HybridClock initialized: now 1790585597359582 us; error 29 us; skew 500 ppm
Sep 28 08:53:17 dist-test-slave-pbjp krb5kdc[25900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585597, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:17.362004 26297 init.cc:377] Logged in from keytab as kudu/127.23.167.195@KRBTEST.COM (short username kudu)
I20260928 08:53:17.362485 26297 webserver.cc:460] Webserver started at http://127.23.167.195:33163/ using document root <none> and password file <none>
I20260928 08:53:17.362704 26297 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:17.362749 26297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:17.362869 26297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:17.363932 26297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-2/data/instance:
uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf"
format_stamp: "Formatted at 2026-09-28 08:53:17 on dist-test-slave-pbjp"
server_key: "6ad54cadd859f5d2c07529e7bfcf00ff"
server_key_iv: "c338e4a5520d55c38d78106aacdbe069"
server_key_version: "encryptionkey@0"
I20260928 08:53:17.364266 26297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance:
uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf"
format_stamp: "Formatted at 2026-09-28 08:53:17 on dist-test-slave-pbjp"
server_key: "6ad54cadd859f5d2c07529e7bfcf00ff"
server_key_iv: "c338e4a5520d55c38d78106aacdbe069"
server_key_version: "encryptionkey@0"
I20260928 08:53:17.365581 26297 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:53:17.366238 26312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:17.366416 26297 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:53:17.366504 26297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf"
format_stamp: "Formatted at 2026-09-28 08:53:17 on dist-test-slave-pbjp"
server_key: "6ad54cadd859f5d2c07529e7bfcf00ff"
server_key_iv: "c338e4a5520d55c38d78106aacdbe069"
server_key_version: "encryptionkey@0"
I20260928 08:53:17.366565 26297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:17.384838 26297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:17.386876 26297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:17.387014 26297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:17.387580 26297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:17.387619 26297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:17.387650 26297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:17.387666 26297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:53:17 dist-test-slave-pbjp krb5kdc[25900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585597, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:17.394564 26297 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:45937
I20260928 08:53:17.394642 26427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:45937 every 8 connection(s)
I20260928 08:53:17.394965 26297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
I20260928 08:53:17.397329 25994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.387961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:56119 (local address 127.23.167.254:46595)
0928 08:53:17.388027 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:17.388029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:17.388228 (+ 199us) server_negotiation.cc:407] Connection header received
0928 08:53:17.388747 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:17.388748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:17.388760 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:17.388846 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:17.389499 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.390274 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.391803 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.392015 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.392072 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.394482 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:17.394483 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:17.394484 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:17.394492 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:17.395927 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:17.396307 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:17.396308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:17.396308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:17.396361 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:17.396633 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:17.396634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:17.396634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:17.396758 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:17.396818 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:17.397152 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 08:53:17.397240 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 08:53:17.397509 26321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.387943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:46595 (local address 127.23.167.195:56119)
0928 08:53:17.388099 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:17.388141 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:17.388651 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:17.388919 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:17.388923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:17.389221 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:17.389423 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.389428 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.390392 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.390393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.391702 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.391707 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.392095 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.392096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.392259 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.392719 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:17.392729 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:17.394391 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:17.396003 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:17.396008 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:17.396011 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:17.396221 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:17.396414 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:17.396415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:17.396415 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:17.396523 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:17.396816 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:17.396823 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:17.397005 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 08:53:17.397127 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 08:53:17.397289 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260928 08:53:17.397974 26428 heartbeater.cc:347] Connected to a master server at 127.23.167.254:46595
I20260928 08:53:17.398080 26428 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:17.398258 26428 heartbeater.cc:511] Master 127.23.167.254:46595 requested a full tablet report, sending...
I20260928 08:53:17.398577 25945 ts_manager.cc:194] Registered new tserver with Master: 6f1c68e27c0b4a0d96e01f5bf1931ddf (127.23.167.195:45937)
I20260928 08:53:17.399032 25945 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.195@KRBTEST.COM'} at 127.23.167.195:56119
I20260928 08:53:17.403805 25994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.398989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:41191 (local address 127.23.167.254:46595)
0928 08:53:17.399036 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:17.399038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:17.399142 (+ 104us) server_negotiation.cc:407] Connection header received
0928 08:53:17.399233 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:17.399234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:17.399242 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:17.399322 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:17.399584 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.400114 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.400851 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.401029 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.401175 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.401983 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:17.401985 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:17.401985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:17.401991 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:17.403022 (+ 1031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:17.403288 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:17.403289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:17.403289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:17.403331 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:17.403466 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:17.403467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:17.403467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:17.403553 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:17.403598 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:17.403692 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 08:53:17.403746 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 08:53:17.403815 26434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.398942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:46595 (local address 127.23.167.195:41191)
0928 08:53:17.399088 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:17.399097 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:17.399164 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:17.399302 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:17.399303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:17.399464 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:17.399520 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.399521 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.400206 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.400207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.400787 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.400789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.401176 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.401177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.401236 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.401653 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:17.401657 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:17.401909 (+ 252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:17.403086 (+ 1177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:17.403087 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:17.403088 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:17.403236 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:17.403366 (+ 130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:17.403366 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:17.403367 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:17.403432 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:17.403595 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:17.403595 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:17.403629 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 08:53:17.403670 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 08:53:17.403744 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":63,"threads_started":1}
I20260928 08:53:17.404417 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 26297
I20260928 08:53:17.404492 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance
I20260928 08:53:17.404664 24223 external_mini_cluster.cc:1428] Setting key 40ff6687f273dff8ea5f03cd95e52ad5
I20260928 08:53:17.405519 24223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 08:53:17 dist-test-slave-pbjp krb5kdc[25900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 08:53:17 dist-test-slave-pbjp krb5kdc[25900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:17.426513 25994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.420213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39534 (local address 127.23.167.254:46595)
0928 08:53:17.420270 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:17.420272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:17.420280 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:53:17.420305 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:17.420306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:17.420315 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:17.420363 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:17.420889 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.421467 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.422167 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.422347 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.422436 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.424550 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:17.424551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:17.424551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:17.424558 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:17.425676 (+ 1118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:17.425975 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:17.425976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:17.425976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:17.426020 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:17.426148 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:17.426148 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:17.426148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:17.426244 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:17.426313 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:17.426387 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 08:53:17.426447 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 08:53:17.427956 25945 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:53:17.429553 25945 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:53:17.436050 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.431913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34124 (local address 127.23.167.193:42119)
0928 08:53:17.432082 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:17.432087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:17.432257 (+ 170us) server_negotiation.cc:407] Connection header received
0928 08:53:17.432377 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:17.432380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:17.432397 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:17.432752 (+ 355us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:53:17.432927 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.433697 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.435117 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.435638 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.435704 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.435763 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:17.435897 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 08:53:17.435934 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":64,"threads_started":1}
I20260928 08:53:17.436053 26446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.431798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:42119 (local address 127.0.0.1:34124)
0928 08:53:17.432148 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:17.432189 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:17.432268 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:17.432824 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:17.432825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:17.432827 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:53:17.432861 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.432862 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.433771 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.433772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.435019 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.435022 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.435715 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.435717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.435794 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.435809 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 08:53:17.435876 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 08:53:17.435934 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":91,"threads_started":1}
I20260928 08:53:17.436233 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.431856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42570 (local address 127.23.167.194:33449)
0928 08:53:17.432082 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:17.432087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:17.432389 (+ 302us) server_negotiation.cc:407] Connection header received
0928 08:53:17.432437 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:17.432439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:17.432453 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:17.432523 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:53:17.432688 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.433902 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.434974 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.435652 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.435723 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.435774 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:17.436094 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 08:53:17.436140 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":95,"threads_started":1}
I20260928 08:53:17.436425 26450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.432078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48534 (local address 127.23.167.195:45937)
0928 08:53:17.432251 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:17.432255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:17.432491 (+ 236us) server_negotiation.cc:407] Connection header received
0928 08:53:17.432536 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:17.432539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:17.432553 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:17.432624 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:53:17.432759 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.433907 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.435180 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.435877 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.435940 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.435986 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:17.436292 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 08:53:17.436346 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":85,"threads_started":1}
I20260928 08:53:17.436527 26445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.431759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:33449 (local address 127.0.0.1:42570)
0928 08:53:17.432147 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:17.432198 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:17.432271 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:17.432527 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:17.432530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:17.432532 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:53:17.432605 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.432607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.433998 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.433999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.434883 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.434886 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.435809 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.435810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.435873 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.435877 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:53:17.435932 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 08:53:17.435978 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":8,"thread_start_us":67,"threads_started":1}
I20260928 08:53:17.436220 26449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.431971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:45937 (local address 127.0.0.1:48534)
0928 08:53:17.432393 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:17.432401 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:17.432442 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:17.432633 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:17.432634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:17.432636 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:53:17.432693 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.432695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.433981 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.433982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.435104 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.435107 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.436016 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.436017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.436076 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.436079 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:53:17.436123 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 08:53:17.436155 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":29,"thread_start_us":71,"threads_started":1}
I20260928 08:53:17.437263 26082 tablet_service.cc:1467] Processing CreateTablet for tablet d12942cd9c6a47878b1d09fe397e2a49 (DEFAULT_TABLE table=test-table [id=fd8f1f4989864b979962728ec4b37d84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:53:17.437288 26222 tablet_service.cc:1467] Processing CreateTablet for tablet d12942cd9c6a47878b1d09fe397e2a49 (DEFAULT_TABLE table=test-table [id=fd8f1f4989864b979962728ec4b37d84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:53:17.437300 26346 tablet_service.cc:1467] Processing CreateTablet for tablet d12942cd9c6a47878b1d09fe397e2a49 (DEFAULT_TABLE table=test-table [id=fd8f1f4989864b979962728ec4b37d84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:53:17.437633 26082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d12942cd9c6a47878b1d09fe397e2a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:17.437634 26346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d12942cd9c6a47878b1d09fe397e2a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:17.437634 26222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d12942cd9c6a47878b1d09fe397e2a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:17.440660 26451 tablet_bootstrap.cc:492] T d12942cd9c6a47878b1d09fe397e2a49 P 10d5981ef80e4fc5a634c389c7b512f0: Bootstrap starting.
I20260928 08:53:17.441040 26453 tablet_bootstrap.cc:492] T d12942cd9c6a47878b1d09fe397e2a49 P 8891e45017924cd794b7860438bb8cc0: Bootstrap starting.
I20260928 08:53:17.441097 26452 tablet_bootstrap.cc:492] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Bootstrap starting.
I20260928 08:53:17.441612 26451 tablet_bootstrap.cc:654] T d12942cd9c6a47878b1d09fe397e2a49 P 10d5981ef80e4fc5a634c389c7b512f0: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:17.441628 26453 tablet_bootstrap.cc:654] T d12942cd9c6a47878b1d09fe397e2a49 P 8891e45017924cd794b7860438bb8cc0: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:17.441875 26452 tablet_bootstrap.cc:654] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:17.441948 26453 log.cc:824] T d12942cd9c6a47878b1d09fe397e2a49 P 8891e45017924cd794b7860438bb8cc0: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:17.441948 26451 log.cc:824] T d12942cd9c6a47878b1d09fe397e2a49 P 10d5981ef80e4fc5a634c389c7b512f0: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:17.442175 26452 log.cc:824] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:17.442739 26451 tablet_bootstrap.cc:492] T d12942cd9c6a47878b1d09fe397e2a49 P 10d5981ef80e4fc5a634c389c7b512f0: No bootstrap required, opened a new log
I20260928 08:53:17.442799 26451 ts_tablet_manager.cc:1397] T d12942cd9c6a47878b1d09fe397e2a49 P 10d5981ef80e4fc5a634c389c7b512f0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:53:17.442838 26453 tablet_bootstrap.cc:492] T d12942cd9c6a47878b1d09fe397e2a49 P 8891e45017924cd794b7860438bb8cc0: No bootstrap required, opened a new log
I20260928 08:53:17.442902 26453 ts_tablet_manager.cc:1397] T d12942cd9c6a47878b1d09fe397e2a49 P 8891e45017924cd794b7860438bb8cc0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 08:53:17.444397 26453 raft_consensus.cc:348] T d12942cd9c6a47878b1d09fe397e2a49 P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } } peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } }
I20260928 08:53:17.444530 26453 raft_consensus.cc:374] T d12942cd9c6a47878b1d09fe397e2a49 P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:17.444550 26453 raft_consensus.cc:729] T d12942cd9c6a47878b1d09fe397e2a49 P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8891e45017924cd794b7860438bb8cc0, State: Initialized, Role: FOLLOWER
I20260928 08:53:17.444568 26451 raft_consensus.cc:348] T d12942cd9c6a47878b1d09fe397e2a49 P 10d5981ef80e4fc5a634c389c7b512f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } } peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } }
I20260928 08:53:17.444650 26453 consensus_queue.cc:260] T d12942cd9c6a47878b1d09fe397e2a49 P 8891e45017924cd794b7860438bb8cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } } peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } }
I20260928 08:53:17.444720 26451 raft_consensus.cc:374] T d12942cd9c6a47878b1d09fe397e2a49 P 10d5981ef80e4fc5a634c389c7b512f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:17.444754 26451 raft_consensus.cc:729] T d12942cd9c6a47878b1d09fe397e2a49 P 10d5981ef80e4fc5a634c389c7b512f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10d5981ef80e4fc5a634c389c7b512f0, State: Initialized, Role: FOLLOWER
I20260928 08:53:17.444906 26453 ts_tablet_manager.cc:1428] T d12942cd9c6a47878b1d09fe397e2a49 P 8891e45017924cd794b7860438bb8cc0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 08:53:17.444923 26288 heartbeater.cc:503] Master 127.23.167.254:46595 was elected leader, sending a full tablet report...
I20260928 08:53:17.445189 26451 consensus_queue.cc:260] T d12942cd9c6a47878b1d09fe397e2a49 P 10d5981ef80e4fc5a634c389c7b512f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } } peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } }
I20260928 08:53:17.445312 26452 tablet_bootstrap.cc:492] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf: No bootstrap required, opened a new log
I20260928 08:53:17.445381 26452 ts_tablet_manager.cc:1397] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 08:53:17.445458 26451 ts_tablet_manager.cc:1428] T d12942cd9c6a47878b1d09fe397e2a49 P 10d5981ef80e4fc5a634c389c7b512f0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:53:17.445580 26148 heartbeater.cc:503] Master 127.23.167.254:46595 was elected leader, sending a full tablet report...
I20260928 08:53:17.447078 26452 raft_consensus.cc:348] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } } peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } }
I20260928 08:53:17.447201 26452 raft_consensus.cc:374] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:17.447221 26452 raft_consensus.cc:729] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f1c68e27c0b4a0d96e01f5bf1931ddf, State: Initialized, Role: FOLLOWER
I20260928 08:53:17.447342 26452 consensus_queue.cc:260] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } } peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } }
I20260928 08:53:17.447589 26452 ts_tablet_manager.cc:1428] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:53:17.447643 26428 heartbeater.cc:503] Master 127.23.167.254:46595 was elected leader, sending a full tablet report...
W20260928 08:53:17.495249 26289 tablet.cc:2369] T d12942cd9c6a47878b1d09fe397e2a49 P 8891e45017924cd794b7860438bb8cc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:53:17.581355 26149 tablet.cc:2369] T d12942cd9c6a47878b1d09fe397e2a49 P 10d5981ef80e4fc5a634c389c7b512f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:53:17.645810 26429 tablet.cc:2369] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:53:17.648995 26459 raft_consensus.cc:482] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:53:17.649128 26459 raft_consensus.cc:504] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } } peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } }
I20260928 08:53:17.649466 26459 leader_election.cc:288] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8891e45017924cd794b7860438bb8cc0 (127.23.167.194:33449), 10d5981ef80e4fc5a634c389c7b512f0 (127.23.167.193:42119)
I20260928 08:53:17.652254 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.649705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:43695 (local address 127.23.167.194:33449)
0928 08:53:17.649750 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:17.649753 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:17.650019 (+ 266us) server_negotiation.cc:407] Connection header received
0928 08:53:17.650048 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:17.650049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:17.650059 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:17.650107 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:53:17.650239 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.650735 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.651602 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.651943 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.652018 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.652027 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:17.652152 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 08:53:17.652187 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 08:53:17.652259 26321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.649649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:42119 (local address 127.23.167.195:52329)
0928 08:53:17.649692 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:17.649702 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:17.649747 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:17.649941 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:17.649942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:17.649945 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:53:17.650010 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.650011 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.650704 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.650705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.651470 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.651473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.652003 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.652004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.652081 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.652084 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:53:17.652130 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 08:53:17.652170 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 08:53:17.652257 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.649761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:52329 (local address 127.23.167.193:42119)
0928 08:53:17.649801 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:17.649802 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:17.649809 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:53:17.649835 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:17.649836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:17.649843 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:17.649908 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:53:17.650097 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.650631 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.651559 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.651925 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.651979 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.651986 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:17.652150 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 08:53:17.652190 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 08:53:17.652282 26460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.649649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:33449 (local address 127.23.167.195:43695)
0928 08:53:17.649956 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:17.649963 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:17.650012 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:17.650110 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:17.650111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:17.650112 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:53:17.650180 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.650182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.650806 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.650807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.651511 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.651513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.652025 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.652026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.652092 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.652095 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:53:17.652141 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 08:53:17.652183 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":143,"thread_start_us":91,"threads_started":1}
I20260928 08:53:17.652642 26102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d12942cd9c6a47878b1d09fe397e2a49" candidate_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10d5981ef80e4fc5a634c389c7b512f0" is_pre_election: true
I20260928 08:53:17.652796 26102 raft_consensus.cc:2454] T d12942cd9c6a47878b1d09fe397e2a49 P 10d5981ef80e4fc5a634c389c7b512f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f1c68e27c0b4a0d96e01f5bf1931ddf in term 0.
I20260928 08:53:17.652907 26242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d12942cd9c6a47878b1d09fe397e2a49" candidate_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8891e45017924cd794b7860438bb8cc0" is_pre_election: true
I20260928 08:53:17.653029 26314 leader_election.cc:302] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10d5981ef80e4fc5a634c389c7b512f0, 6f1c68e27c0b4a0d96e01f5bf1931ddf; no voters:
I20260928 08:53:17.653128 26242 raft_consensus.cc:2454] T d12942cd9c6a47878b1d09fe397e2a49 P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f1c68e27c0b4a0d96e01f5bf1931ddf in term 0.
I20260928 08:53:17.653169 26459 raft_consensus.cc:2781] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:53:17.653223 26459 raft_consensus.cc:482] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:53:17.653244 26459 raft_consensus.cc:3037] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:17.654150 26459 raft_consensus.cc:504] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } } peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } }
I20260928 08:53:17.654268 26459 leader_election.cc:288] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [CANDIDATE]: Term 1 election: Requested vote from peers 8891e45017924cd794b7860438bb8cc0 (127.23.167.194:33449), 10d5981ef80e4fc5a634c389c7b512f0 (127.23.167.193:42119)
I20260928 08:53:17.654402 26102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d12942cd9c6a47878b1d09fe397e2a49" candidate_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10d5981ef80e4fc5a634c389c7b512f0"
I20260928 08:53:17.654417 26242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d12942cd9c6a47878b1d09fe397e2a49" candidate_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8891e45017924cd794b7860438bb8cc0"
I20260928 08:53:17.654480 26242 raft_consensus.cc:3037] T d12942cd9c6a47878b1d09fe397e2a49 P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:17.654484 26102 raft_consensus.cc:3037] T d12942cd9c6a47878b1d09fe397e2a49 P 10d5981ef80e4fc5a634c389c7b512f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:17.655117 26242 raft_consensus.cc:2454] T d12942cd9c6a47878b1d09fe397e2a49 P 8891e45017924cd794b7860438bb8cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f1c68e27c0b4a0d96e01f5bf1931ddf in term 1.
I20260928 08:53:17.655202 26102 raft_consensus.cc:2454] T d12942cd9c6a47878b1d09fe397e2a49 P 10d5981ef80e4fc5a634c389c7b512f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f1c68e27c0b4a0d96e01f5bf1931ddf in term 1.
I20260928 08:53:17.655277 26315 leader_election.cc:302] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f1c68e27c0b4a0d96e01f5bf1931ddf, 8891e45017924cd794b7860438bb8cc0; no voters:
I20260928 08:53:17.655354 26459 raft_consensus.cc:2781] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:53:17.655474 26459 raft_consensus.cc:686] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 1 LEADER]: Becoming Leader. State: Replica: 6f1c68e27c0b4a0d96e01f5bf1931ddf, State: Running, Role: LEADER
I20260928 08:53:17.655560 26459 consensus_queue.cc:237] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } } peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } }
I20260928 08:53:17.656374 25945 catalog_manager.cc:5495] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f1c68e27c0b4a0d96e01f5bf1931ddf (127.23.167.195). New cstate: current_term: 1 leader_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } health_report { overall_health: UNKNOWN } } }
Sep 28 08:53:17 dist-test-slave-pbjp krb5kdc[25900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585596, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:17.683944 26468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.677719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:46595 (local address 127.0.0.1:39548)
0928 08:53:17.677856 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:17.677872 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:17.677945 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:17.678069 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:17.678070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:17.678374 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:17.678437 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.678439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.679065 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.679066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.679642 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.679644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.680048 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.680048 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.680099 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.680581 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:17.680585 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:17.681853 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:17.683014 (+ 1161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:17.683022 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:17.683024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:17.683200 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:17.683356 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:17.683356 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:17.683357 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:17.683413 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:17.683610 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:17.683617 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:17.683660 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 08:53:17.683717 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 08:53:17.683829 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260928 08:53:17.683993 25994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.677771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39548 (local address 127.23.167.254:46595)
0928 08:53:17.677822 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:17.677824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:17.677932 (+ 108us) server_negotiation.cc:407] Connection header received
0928 08:53:17.677999 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:17.678000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:17.678010 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:17.678058 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:17.678508 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.678997 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.679731 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.679971 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.680049 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.681929 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:17.681930 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:17.681931 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:17.681938 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:17.682930 (+ 992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:17.683258 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:17.683259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:17.683260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:17.683306 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:17.683459 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:17.683460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:17.683460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:17.683559 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:17.683612 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:17.683742 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 08:53:17.683895 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":25}
I20260928 08:53:17.685994 25944 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:39548:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 08:53:17.686173 25944 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:53:17.687578 26082 tablet_service.cc:1467] Processing CreateTablet for tablet f2c9d0bcbabb4e9aa109e05e9851c9ef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ff760611afa34eeeb77dabd521398ecd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:53:17.687592 26346 tablet_service.cc:1467] Processing CreateTablet for tablet f2c9d0bcbabb4e9aa109e05e9851c9ef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ff760611afa34eeeb77dabd521398ecd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:53:17.687736 26082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2c9d0bcbabb4e9aa109e05e9851c9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:17.687753 26346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2c9d0bcbabb4e9aa109e05e9851c9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:17.688900 26452 tablet_bootstrap.cc:492] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Bootstrap starting.
I20260928 08:53:17.688958 26451 tablet_bootstrap.cc:492] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 10d5981ef80e4fc5a634c389c7b512f0: Bootstrap starting.
I20260928 08:53:17.689103 26222 tablet_service.cc:1467] Processing CreateTablet for tablet f2c9d0bcbabb4e9aa109e05e9851c9ef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ff760611afa34eeeb77dabd521398ecd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:53:17.689249 26222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2c9d0bcbabb4e9aa109e05e9851c9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:17.689337 26452 tablet_bootstrap.cc:654] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:17.689340 26451 tablet_bootstrap.cc:654] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 10d5981ef80e4fc5a634c389c7b512f0: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:17.689913 26451 tablet_bootstrap.cc:492] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 10d5981ef80e4fc5a634c389c7b512f0: No bootstrap required, opened a new log
I20260928 08:53:17.689956 26451 ts_tablet_manager.cc:1397] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 10d5981ef80e4fc5a634c389c7b512f0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 08:53:17.690101 26451 raft_consensus.cc:348] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 10d5981ef80e4fc5a634c389c7b512f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:17.690160 26451 raft_consensus.cc:374] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 10d5981ef80e4fc5a634c389c7b512f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:17.690173 26451 raft_consensus.cc:729] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 10d5981ef80e4fc5a634c389c7b512f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10d5981ef80e4fc5a634c389c7b512f0, State: Initialized, Role: FOLLOWER
I20260928 08:53:17.690207 26451 consensus_queue.cc:260] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 10d5981ef80e4fc5a634c389c7b512f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:17.690306 26451 ts_tablet_manager.cc:1428] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 10d5981ef80e4fc5a634c389c7b512f0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:17.690351 26458 tablet_replica.cc:439] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 10d5981ef80e4fc5a634c389c7b512f0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } } }
I20260928 08:53:17.690436 26458 tablet_replica.cc:442] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 10d5981ef80e4fc5a634c389c7b512f0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:53:17.690621 26453 tablet_bootstrap.cc:492] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0: Bootstrap starting.
I20260928 08:53:17.691030 26452 tablet_bootstrap.cc:492] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 6f1c68e27c0b4a0d96e01f5bf1931ddf: No bootstrap required, opened a new log
I20260928 08:53:17.691072 26452 ts_tablet_manager.cc:1397] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:53:17.691200 26453 tablet_bootstrap.cc:654] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:17.691198 26452 raft_consensus.cc:348] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:17.691257 26452 raft_consensus.cc:374] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:17.691283 26452 raft_consensus.cc:729] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f1c68e27c0b4a0d96e01f5bf1931ddf, State: Initialized, Role: FOLLOWER
I20260928 08:53:17.691330 26452 consensus_queue.cc:260] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 6f1c68e27c0b4a0d96e01f5bf1931ddf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:17.691417 26452 ts_tablet_manager.cc:1428] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:17.691478 26459 tablet_replica.cc:439] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 6f1c68e27c0b4a0d96e01f5bf1931ddf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } } }
I20260928 08:53:17.691550 26459 tablet_replica.cc:442] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 6f1c68e27c0b4a0d96e01f5bf1931ddf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:53:17.691887 26453 tablet_bootstrap.cc:492] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0: No bootstrap required, opened a new log
I20260928 08:53:17.691936 26453 ts_tablet_manager.cc:1397] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 08:53:17.692063 26453 raft_consensus.cc:348] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:17.692108 26453 raft_consensus.cc:374] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:17.692119 26453 raft_consensus.cc:729] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8891e45017924cd794b7860438bb8cc0, State: Initialized, Role: FOLLOWER
I20260928 08:53:17.692150 26453 consensus_queue.cc:260] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:17.692224 26453 ts_tablet_manager.cc:1428] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:17.692317 26457 tablet_replica.cc:439] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } } }
I20260928 08:53:17.692390 26457 tablet_replica.cc:442] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:53:17.848863 26457 raft_consensus.cc:482] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:53:17.848976 26457 raft_consensus.cc:504] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:17.849371 26457 leader_election.cc:288] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10d5981ef80e4fc5a634c389c7b512f0 (127.23.167.193:42119), 6f1c68e27c0b4a0d96e01f5bf1931ddf (127.23.167.195:45937)
I20260928 08:53:17.852114 26450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.849659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:60937 (local address 127.23.167.195:45937)
0928 08:53:17.849709 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:17.849711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:17.849788 (+ 77us) server_negotiation.cc:407] Connection header received
0928 08:53:17.849861 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:17.849862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:17.849871 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:17.849911 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:53:17.850051 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.850521 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.851419 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.851793 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.851859 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.851868 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:17.852014 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 08:53:17.852049 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 08:53:17.852135 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.849714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:50915 (local address 127.23.167.193:42119)
0928 08:53:17.849775 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:17.849777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:17.849870 (+ 93us) server_negotiation.cc:407] Connection header received
0928 08:53:17.849953 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:17.849954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:17.849963 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:17.850009 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:53:17.850136 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.850593 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.851463 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.851802 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.851862 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.851871 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:17.852032 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 08:53:17.852065 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 08:53:17.852162 26473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.849572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:45937 (local address 127.23.167.194:60937)
0928 08:53:17.849706 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:17.849717 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:17.849796 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:17.849919 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:17.849920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:17.849922 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:53:17.849994 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.849996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.850589 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.850590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.851332 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.851334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.851864 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.851865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.851952 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.851956 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:53:17.851998 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 08:53:17.852040 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":2,"thread_start_us":62,"threads_started":1}
I20260928 08:53:17.852162 26472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.849572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:42119 (local address 127.23.167.194:50915)
0928 08:53:17.849800 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:17.849810 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:17.849870 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:17.850015 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:17.850016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:17.850018 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:53:17.850081 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.850083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.850660 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.850661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.851360 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.851362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.851868 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.851869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.851967 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.851971 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:53:17.852016 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 08:53:17.852058 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":37,"thread_start_us":64,"threads_started":1}
I20260928 08:53:17.852458 26382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c9d0bcbabb4e9aa109e05e9851c9ef" candidate_uuid: "8891e45017924cd794b7860438bb8cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" is_pre_election: true
I20260928 08:53:17.852586 26382 raft_consensus.cc:2454] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8891e45017924cd794b7860438bb8cc0 in term 0.
I20260928 08:53:17.852557 26102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c9d0bcbabb4e9aa109e05e9851c9ef" candidate_uuid: "8891e45017924cd794b7860438bb8cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10d5981ef80e4fc5a634c389c7b512f0" is_pre_election: true
I20260928 08:53:17.852643 26102 raft_consensus.cc:2454] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 10d5981ef80e4fc5a634c389c7b512f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8891e45017924cd794b7860438bb8cc0 in term 0.
I20260928 08:53:17.852771 26175 leader_election.cc:302] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f1c68e27c0b4a0d96e01f5bf1931ddf, 8891e45017924cd794b7860438bb8cc0; no voters:
I20260928 08:53:17.852906 26457 raft_consensus.cc:2781] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:53:17.852957 26457 raft_consensus.cc:482] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:53:17.852972 26457 raft_consensus.cc:3037] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:17.853621 26457 raft_consensus.cc:504] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:17.853744 26457 leader_election.cc:288] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0 [CANDIDATE]: Term 1 election: Requested vote from peers 10d5981ef80e4fc5a634c389c7b512f0 (127.23.167.193:42119), 6f1c68e27c0b4a0d96e01f5bf1931ddf (127.23.167.195:45937)
I20260928 08:53:17.853871 26102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c9d0bcbabb4e9aa109e05e9851c9ef" candidate_uuid: "8891e45017924cd794b7860438bb8cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10d5981ef80e4fc5a634c389c7b512f0"
I20260928 08:53:17.853963 26102 raft_consensus.cc:3037] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 10d5981ef80e4fc5a634c389c7b512f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:17.854583 26102 raft_consensus.cc:2454] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 10d5981ef80e4fc5a634c389c7b512f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8891e45017924cd794b7860438bb8cc0 in term 1.
I20260928 08:53:17.854764 26174 leader_election.cc:302] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10d5981ef80e4fc5a634c389c7b512f0, 8891e45017924cd794b7860438bb8cc0; no voters:
I20260928 08:53:17.854771 26382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c9d0bcbabb4e9aa109e05e9851c9ef" candidate_uuid: "8891e45017924cd794b7860438bb8cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf"
I20260928 08:53:17.854838 26382 raft_consensus.cc:3037] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:17.854866 26457 raft_consensus.cc:2781] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:53:17.854996 26457 raft_consensus.cc:686] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0 [term 1 LEADER]: Becoming Leader. State: Replica: 8891e45017924cd794b7860438bb8cc0, State: Running, Role: LEADER
I20260928 08:53:17.855096 26457 consensus_queue.cc:237] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:17.855643 26382 raft_consensus.cc:2454] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8891e45017924cd794b7860438bb8cc0 in term 1.
I20260928 08:53:17.855717 26474 tablet_replica.cc:439] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0: TxnStatusTablet state changed. Reason: New leader 8891e45017924cd794b7860438bb8cc0. Latest consensus state: current_term: 1 leader_uuid: "8891e45017924cd794b7860438bb8cc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } } }
I20260928 08:53:17.855798 26474 tablet_replica.cc:442] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:53:17.855983 26476 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 08:53:17.856091 26476 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 08:53:17.855939 25944 catalog_manager.cc:5495] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8891e45017924cd794b7860438bb8cc0 (127.23.167.194). New cstate: current_term: 1 leader_uuid: "8891e45017924cd794b7860438bb8cc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } health_report { overall_health: UNKNOWN } } }
I20260928 08:53:17.873826 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.871506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42578 (local address 127.23.167.194:33449)
0928 08:53:17.871566 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:17.871568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:17.871578 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:53:17.871635 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:17.871635 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:17.871656 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:17.871711 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:53:17.871836 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.872331 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.873045 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.873285 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.873340 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.873507 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:53:17.873616 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:53:17.873685 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:17.873723 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 08:53:17.873746 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 08:53:17.873834 26468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.871430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:33449 (local address 127.0.0.1:42578)
0928 08:53:17.871494 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:17.871506 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:17.871555 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:17.871719 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:17.871720 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:17.871721 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:53:17.871774 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.871776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.872399 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.872400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.872971 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.872974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.873345 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.873346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.873414 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.873431 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:53:17.873668 (+ 237us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:53:17.873676 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 08:53:17.873721 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 08:53:17.873767 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 08:53:17.875569 26382 raft_consensus.cc:1261] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 1 FOLLOWER]: Refusing update from remote peer 8891e45017924cd794b7860438bb8cc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:53:17.875665 26102 raft_consensus.cc:1261] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 10d5981ef80e4fc5a634c389c7b512f0 [term 1 FOLLOWER]: Refusing update from remote peer 8891e45017924cd794b7860438bb8cc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:53:17.875859 26474 consensus_queue.cc:1035] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:53:17.876001 26457 consensus_queue.cc:1035] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:53:17.877192 26459 tablet_replica.cc:439] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 6f1c68e27c0b4a0d96e01f5bf1931ddf: TxnStatusTablet state changed. Reason: New leader 8891e45017924cd794b7860438bb8cc0. Latest consensus state: current_term: 1 leader_uuid: "8891e45017924cd794b7860438bb8cc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } } }
I20260928 08:53:17.877285 26459 tablet_replica.cc:442] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 6f1c68e27c0b4a0d96e01f5bf1931ddf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:53:17.877251 26458 tablet_replica.cc:439] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 10d5981ef80e4fc5a634c389c7b512f0: TxnStatusTablet state changed. Reason: New leader 8891e45017924cd794b7860438bb8cc0. Latest consensus state: current_term: 1 leader_uuid: "8891e45017924cd794b7860438bb8cc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } } }
I20260928 08:53:17.877336 26458 tablet_replica.cc:442] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 10d5981ef80e4fc5a634c389c7b512f0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:53:17.878459 26479 mvcc.cc:205] Tried to move back new op lower bound from 7334238608895975424 to 7334238608815017984. Current Snapshot: MvccSnapshot[applied={T|T < 7334238608895975424}]
I20260928 08:53:17.878541 26457 tablet_replica.cc:439] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8891e45017924cd794b7860438bb8cc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } } }
I20260928 08:53:17.878631 26457 tablet_replica.cc:442] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:53:17.879006 26458 tablet_replica.cc:439] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 10d5981ef80e4fc5a634c389c7b512f0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8891e45017924cd794b7860438bb8cc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } } }
I20260928 08:53:17.879088 26458 tablet_replica.cc:442] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 10d5981ef80e4fc5a634c389c7b512f0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:53:17.881642 26457 tablet_replica.cc:439] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8891e45017924cd794b7860438bb8cc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } } }
I20260928 08:53:17.881728 26457 tablet_replica.cc:442] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 8891e45017924cd794b7860438bb8cc0: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:53:17.881984 26459 tablet_replica.cc:439] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 6f1c68e27c0b4a0d96e01f5bf1931ddf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8891e45017924cd794b7860438bb8cc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } } }
I20260928 08:53:17.882068 26459 tablet_replica.cc:442] T f2c9d0bcbabb4e9aa109e05e9851c9ef P 6f1c68e27c0b4a0d96e01f5bf1931ddf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:53:17.882170 26481 mvcc.cc:205] Tried to move back new op lower bound from 7334238608895975424 to 7334238608815017984. Current Snapshot: MvccSnapshot[applied={T|T < 7334238608895975424}]
I20260928 08:53:17.883703 26450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.880873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48536 (local address 127.23.167.195:45937)
0928 08:53:17.880915 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:17.880917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:17.880979 (+ 62us) server_negotiation.cc:407] Connection header received
0928 08:53:17.881085 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:17.881086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:17.881106 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:17.881172 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:53:17.881276 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.881920 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.882880 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.883193 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.883261 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.883423 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:53:17.883518 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:53:17.883576 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:17.883633 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 08:53:17.883653 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 08:53:17.888108 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.885969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:52131 (local address 127.23.167.194:33449)
0928 08:53:17.886011 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:17.886013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:17.886021 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:53:17.886042 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:17.886043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:17.886052 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:17.886101 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:53:17.886219 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.886688 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.887392 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.887661 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.887713 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.887892 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:53:17.887924 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:53:17.887968 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:17.888018 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 08:53:17.888037 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 08:53:17.888116 26434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.885881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:33449 (local address 127.23.167.195:52131)
0928 08:53:17.885951 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:17.885962 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:17.886000 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:17.886105 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:17.886105 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:17.886106 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:53:17.886161 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.886163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.886757 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.886758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.887319 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.887321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.887733 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.887734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.887798 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.887811 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:53:17.887974 (+ 163us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:53:17.887975 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 08:53:17.888017 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 08:53:17.888057 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 08:53:17.892413 26242 raft_consensus.cc:1261] T d12942cd9c6a47878b1d09fe397e2a49 P 8891e45017924cd794b7860438bb8cc0 [term 1 FOLLOWER]: Refusing update from remote peer 6f1c68e27c0b4a0d96e01f5bf1931ddf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:53:17.892590 26102 raft_consensus.cc:1261] T d12942cd9c6a47878b1d09fe397e2a49 P 10d5981ef80e4fc5a634c389c7b512f0 [term 1 FOLLOWER]: Refusing update from remote peer 6f1c68e27c0b4a0d96e01f5bf1931ddf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:53:17.892686 26461 consensus_queue.cc:1035] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:53:17.892784 26459 consensus_queue.cc:1035] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [LEADER]: Connected to new peer: Peer: permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:53:17.901160 26450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.898648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:33625 (local address 127.23.167.195:45937)
0928 08:53:17.898688 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:17.898690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:17.898822 (+ 132us) server_negotiation.cc:407] Connection header received
0928 08:53:17.898879 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:17.898879 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:17.898888 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:17.898942 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:53:17.899077 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.899555 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.900346 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.900627 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.900688 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.900884 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:53:17.900919 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:53:17.900964 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:17.901042 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 08:53:17.901085 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 08:53:17.901196 26493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:17.898611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:45937 (local address 127.23.167.194:33625)
0928 08:53:17.898763 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:17.898774 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:17.898834 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:17.898966 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:17.898967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:17.898968 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:53:17.899024 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.899026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.899627 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.899628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.900213 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:17.900215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:17.900698 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:17.900699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:17.900782 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:17.900799 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:53:17.900985 (+ 186us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:53:17.900986 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 08:53:17.901039 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 08:53:17.901106 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":75,"threads_started":1}
I20260928 08:53:17.910650 25944 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:53:17.910780 25944 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:53:17.911607 25944 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e718379167694065b3c67180d47b82c0: Sending DeleteTablet for 3 replicas of tablet d12942cd9c6a47878b1d09fe397e2a49
I20260928 08:53:17.911931 26082 tablet_service.cc:1514] Processing DeleteTablet for tablet d12942cd9c6a47878b1d09fe397e2a49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:53:17 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:34124
I20260928 08:53:17.912027 26222 tablet_service.cc:1514] Processing DeleteTablet for tablet d12942cd9c6a47878b1d09fe397e2a49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:53:17 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:42570
I20260928 08:53:17.912391 26346 tablet_service.cc:1514] Processing DeleteTablet for tablet d12942cd9c6a47878b1d09fe397e2a49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:53:17 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:48534
I20260928 08:53:17.912544 25944 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:53:17.912698 25944 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:53:17.912822 26499 tablet_replica.cc:330] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf: stopping tablet replica
I20260928 08:53:17.912974 26499 raft_consensus.cc:2229] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:53:17.913153 26499 raft_consensus.cc:2258] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:53:17.913666 26499 ts_tablet_manager.cc:1905] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:53:17.913822 26497 tablet_replica.cc:330] T d12942cd9c6a47878b1d09fe397e2a49 P 10d5981ef80e4fc5a634c389c7b512f0: stopping tablet replica
I20260928 08:53:17.914179 26346 tablet_service.cc:1467] Processing CreateTablet for tablet 233226cd3a29499c92b848b3ba9cc4c4 (DEFAULT_TABLE table=test-table [id=679a561a9a8148749b25018dce650a4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:53:17.913894 26082 tablet_service.cc:1467] Processing CreateTablet for tablet 233226cd3a29499c92b848b3ba9cc4c4 (DEFAULT_TABLE table=test-table [id=679a561a9a8148749b25018dce650a4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:53:17.914296 26346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 233226cd3a29499c92b848b3ba9cc4c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:17.914386 26082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 233226cd3a29499c92b848b3ba9cc4c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:17.915586 26452 tablet_bootstrap.cc:492] T 233226cd3a29499c92b848b3ba9cc4c4 P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Bootstrap starting.
I20260928 08:53:17.915633 26451 tablet_bootstrap.cc:492] T 233226cd3a29499c92b848b3ba9cc4c4 P 10d5981ef80e4fc5a634c389c7b512f0: Bootstrap starting.
I20260928 08:53:17.915967 26452 tablet_bootstrap.cc:654] T 233226cd3a29499c92b848b3ba9cc4c4 P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:17.916286 26451 tablet_bootstrap.cc:654] T 233226cd3a29499c92b848b3ba9cc4c4 P 10d5981ef80e4fc5a634c389c7b512f0: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:17.916558 26222 tablet_service.cc:1467] Processing CreateTablet for tablet 233226cd3a29499c92b848b3ba9cc4c4 (DEFAULT_TABLE table=test-table [id=679a561a9a8148749b25018dce650a4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:53:17.916674 26222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 233226cd3a29499c92b848b3ba9cc4c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:17.916865 26497 raft_consensus.cc:2229] T d12942cd9c6a47878b1d09fe397e2a49 P 10d5981ef80e4fc5a634c389c7b512f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:53:17.916971 26497 raft_consensus.cc:2258] T d12942cd9c6a47878b1d09fe397e2a49 P 10d5981ef80e4fc5a634c389c7b512f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:53:17.918386 26498 tablet_replica.cc:330] T d12942cd9c6a47878b1d09fe397e2a49 P 8891e45017924cd794b7860438bb8cc0: stopping tablet replica
I20260928 08:53:17.918524 26499 ts_tablet_manager.cc:1918] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:53:17.918618 26499 log.cc:1196] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-2/wal/wals/d12942cd9c6a47878b1d09fe397e2a49
I20260928 08:53:17.918951 26499 ts_tablet_manager.cc:1939] T d12942cd9c6a47878b1d09fe397e2a49 P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Deleting consensus metadata
I20260928 08:53:17.918356 26453 tablet_bootstrap.cc:492] T 233226cd3a29499c92b848b3ba9cc4c4 P 8891e45017924cd794b7860438bb8cc0: Bootstrap starting.
I20260928 08:53:17.919124 26497 ts_tablet_manager.cc:1905] T d12942cd9c6a47878b1d09fe397e2a49 P 10d5981ef80e4fc5a634c389c7b512f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:53:17.919518 26453 tablet_bootstrap.cc:654] T 233226cd3a29499c92b848b3ba9cc4c4 P 8891e45017924cd794b7860438bb8cc0: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:17.920213 25931 catalog_manager.cc:4841] TS 6f1c68e27c0b4a0d96e01f5bf1931ddf (127.23.167.195:45937): tablet d12942cd9c6a47878b1d09fe397e2a49 (table test-table [id=fd8f1f4989864b979962728ec4b37d84]) successfully deleted
I20260928 08:53:17.920377 26498 raft_consensus.cc:2229] T d12942cd9c6a47878b1d09fe397e2a49 P 8891e45017924cd794b7860438bb8cc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:53:17.920493 26498 raft_consensus.cc:2258] T d12942cd9c6a47878b1d09fe397e2a49 P 8891e45017924cd794b7860438bb8cc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:53:17.921057 26498 ts_tablet_manager.cc:1905] T d12942cd9c6a47878b1d09fe397e2a49 P 8891e45017924cd794b7860438bb8cc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:53:17.921366 26497 ts_tablet_manager.cc:1918] T d12942cd9c6a47878b1d09fe397e2a49 P 10d5981ef80e4fc5a634c389c7b512f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:53:17.921453 26497 log.cc:1196] T d12942cd9c6a47878b1d09fe397e2a49 P 10d5981ef80e4fc5a634c389c7b512f0: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-0/wal/wals/d12942cd9c6a47878b1d09fe397e2a49
I20260928 08:53:17.921693 26497 ts_tablet_manager.cc:1939] T d12942cd9c6a47878b1d09fe397e2a49 P 10d5981ef80e4fc5a634c389c7b512f0: Deleting consensus metadata
I20260928 08:53:17.922358 25930 catalog_manager.cc:4841] TS 10d5981ef80e4fc5a634c389c7b512f0 (127.23.167.193:42119): tablet d12942cd9c6a47878b1d09fe397e2a49 (table test-table [id=fd8f1f4989864b979962728ec4b37d84]) successfully deleted
I20260928 08:53:17.922645 26452 tablet_bootstrap.cc:492] T 233226cd3a29499c92b848b3ba9cc4c4 P 6f1c68e27c0b4a0d96e01f5bf1931ddf: No bootstrap required, opened a new log
I20260928 08:53:17.922700 26452 ts_tablet_manager.cc:1397] T 233226cd3a29499c92b848b3ba9cc4c4 P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260928 08:53:17.922827 26498 ts_tablet_manager.cc:1918] T d12942cd9c6a47878b1d09fe397e2a49 P 8891e45017924cd794b7860438bb8cc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:53:17.922847 26452 raft_consensus.cc:348] T 233226cd3a29499c92b848b3ba9cc4c4 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:17.922923 26452 raft_consensus.cc:374] T 233226cd3a29499c92b848b3ba9cc4c4 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:17.922946 26452 raft_consensus.cc:729] T 233226cd3a29499c92b848b3ba9cc4c4 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f1c68e27c0b4a0d96e01f5bf1931ddf, State: Initialized, Role: FOLLOWER
I20260928 08:53:17.922988 26452 consensus_queue.cc:260] T 233226cd3a29499c92b848b3ba9cc4c4 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:17.923076 26452 ts_tablet_manager.cc:1428] T 233226cd3a29499c92b848b3ba9cc4c4 P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 08:53:17.923029 26498 log.cc:1196] T d12942cd9c6a47878b1d09fe397e2a49 P 8891e45017924cd794b7860438bb8cc0: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-1/wal/wals/d12942cd9c6a47878b1d09fe397e2a49
I20260928 08:53:17.923434 26498 ts_tablet_manager.cc:1939] T d12942cd9c6a47878b1d09fe397e2a49 P 8891e45017924cd794b7860438bb8cc0: Deleting consensus metadata
I20260928 08:53:17.923870 25931 catalog_manager.cc:4841] TS 8891e45017924cd794b7860438bb8cc0 (127.23.167.194:33449): tablet d12942cd9c6a47878b1d09fe397e2a49 (table test-table [id=fd8f1f4989864b979962728ec4b37d84]) successfully deleted
I20260928 08:53:17.923920 26453 tablet_bootstrap.cc:492] T 233226cd3a29499c92b848b3ba9cc4c4 P 8891e45017924cd794b7860438bb8cc0: No bootstrap required, opened a new log
I20260928 08:53:17.923959 26453 ts_tablet_manager.cc:1397] T 233226cd3a29499c92b848b3ba9cc4c4 P 8891e45017924cd794b7860438bb8cc0: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260928 08:53:17.924091 26453 raft_consensus.cc:348] T 233226cd3a29499c92b848b3ba9cc4c4 P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:17.924150 26453 raft_consensus.cc:374] T 233226cd3a29499c92b848b3ba9cc4c4 P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:17.924167 26453 raft_consensus.cc:729] T 233226cd3a29499c92b848b3ba9cc4c4 P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8891e45017924cd794b7860438bb8cc0, State: Initialized, Role: FOLLOWER
I20260928 08:53:17.924223 26453 consensus_queue.cc:260] T 233226cd3a29499c92b848b3ba9cc4c4 P 8891e45017924cd794b7860438bb8cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:17.924302 26453 ts_tablet_manager.cc:1428] T 233226cd3a29499c92b848b3ba9cc4c4 P 8891e45017924cd794b7860438bb8cc0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:17.925487 26451 tablet_bootstrap.cc:492] T 233226cd3a29499c92b848b3ba9cc4c4 P 10d5981ef80e4fc5a634c389c7b512f0: No bootstrap required, opened a new log
I20260928 08:53:17.925545 26451 ts_tablet_manager.cc:1397] T 233226cd3a29499c92b848b3ba9cc4c4 P 10d5981ef80e4fc5a634c389c7b512f0: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260928 08:53:17.925683 26451 raft_consensus.cc:348] T 233226cd3a29499c92b848b3ba9cc4c4 P 10d5981ef80e4fc5a634c389c7b512f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:17.925753 26451 raft_consensus.cc:374] T 233226cd3a29499c92b848b3ba9cc4c4 P 10d5981ef80e4fc5a634c389c7b512f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:17.925772 26451 raft_consensus.cc:729] T 233226cd3a29499c92b848b3ba9cc4c4 P 10d5981ef80e4fc5a634c389c7b512f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10d5981ef80e4fc5a634c389c7b512f0, State: Initialized, Role: FOLLOWER
I20260928 08:53:17.925817 26451 consensus_queue.cc:260] T 233226cd3a29499c92b848b3ba9cc4c4 P 10d5981ef80e4fc5a634c389c7b512f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:17.925905 26451 ts_tablet_manager.cc:1428] T 233226cd3a29499c92b848b3ba9cc4c4 P 10d5981ef80e4fc5a634c389c7b512f0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 08:53:17.993103 26459 raft_consensus.cc:482] T 233226cd3a29499c92b848b3ba9cc4c4 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:53:17.993206 26459 raft_consensus.cc:504] T 233226cd3a29499c92b848b3ba9cc4c4 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:17.993360 26459 leader_election.cc:288] T 233226cd3a29499c92b848b3ba9cc4c4 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10d5981ef80e4fc5a634c389c7b512f0 (127.23.167.193:42119), 8891e45017924cd794b7860438bb8cc0 (127.23.167.194:33449)
I20260928 08:53:17.993575 26242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "233226cd3a29499c92b848b3ba9cc4c4" candidate_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8891e45017924cd794b7860438bb8cc0" is_pre_election: true
I20260928 08:53:17.993681 26242 raft_consensus.cc:2454] T 233226cd3a29499c92b848b3ba9cc4c4 P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f1c68e27c0b4a0d96e01f5bf1931ddf in term 0.
I20260928 08:53:17.993639 26102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "233226cd3a29499c92b848b3ba9cc4c4" candidate_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10d5981ef80e4fc5a634c389c7b512f0" is_pre_election: true
I20260928 08:53:17.993749 26102 raft_consensus.cc:2454] T 233226cd3a29499c92b848b3ba9cc4c4 P 10d5981ef80e4fc5a634c389c7b512f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f1c68e27c0b4a0d96e01f5bf1931ddf in term 0.
I20260928 08:53:17.993839 26315 leader_election.cc:302] T 233226cd3a29499c92b848b3ba9cc4c4 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f1c68e27c0b4a0d96e01f5bf1931ddf, 8891e45017924cd794b7860438bb8cc0; no voters:
I20260928 08:53:17.993932 26459 raft_consensus.cc:2781] T 233226cd3a29499c92b848b3ba9cc4c4 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:53:17.993978 26459 raft_consensus.cc:482] T 233226cd3a29499c92b848b3ba9cc4c4 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:53:17.993996 26459 raft_consensus.cc:3037] T 233226cd3a29499c92b848b3ba9cc4c4 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:17.994823 26459 raft_consensus.cc:504] T 233226cd3a29499c92b848b3ba9cc4c4 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:17.994941 26459 leader_election.cc:288] T 233226cd3a29499c92b848b3ba9cc4c4 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [CANDIDATE]: Term 1 election: Requested vote from peers 10d5981ef80e4fc5a634c389c7b512f0 (127.23.167.193:42119), 8891e45017924cd794b7860438bb8cc0 (127.23.167.194:33449)
I20260928 08:53:17.995128 26102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "233226cd3a29499c92b848b3ba9cc4c4" candidate_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10d5981ef80e4fc5a634c389c7b512f0"
I20260928 08:53:17.995190 26102 raft_consensus.cc:3037] T 233226cd3a29499c92b848b3ba9cc4c4 P 10d5981ef80e4fc5a634c389c7b512f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:17.995198 26242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "233226cd3a29499c92b848b3ba9cc4c4" candidate_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8891e45017924cd794b7860438bb8cc0"
I20260928 08:53:17.995270 26242 raft_consensus.cc:3037] T 233226cd3a29499c92b848b3ba9cc4c4 P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:17.995872 26102 raft_consensus.cc:2454] T 233226cd3a29499c92b848b3ba9cc4c4 P 10d5981ef80e4fc5a634c389c7b512f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f1c68e27c0b4a0d96e01f5bf1931ddf in term 1.
I20260928 08:53:17.996008 26314 leader_election.cc:302] T 233226cd3a29499c92b848b3ba9cc4c4 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10d5981ef80e4fc5a634c389c7b512f0, 6f1c68e27c0b4a0d96e01f5bf1931ddf; no voters:
I20260928 08:53:17.996054 26242 raft_consensus.cc:2454] T 233226cd3a29499c92b848b3ba9cc4c4 P 8891e45017924cd794b7860438bb8cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f1c68e27c0b4a0d96e01f5bf1931ddf in term 1.
I20260928 08:53:17.996073 26459 raft_consensus.cc:2781] T 233226cd3a29499c92b848b3ba9cc4c4 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:53:17.996101 26459 raft_consensus.cc:686] T 233226cd3a29499c92b848b3ba9cc4c4 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 1 LEADER]: Becoming Leader. State: Replica: 6f1c68e27c0b4a0d96e01f5bf1931ddf, State: Running, Role: LEADER
I20260928 08:53:17.996138 26459 consensus_queue.cc:237] T 233226cd3a29499c92b848b3ba9cc4c4 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:17.996803 25944 catalog_manager.cc:5495] T 233226cd3a29499c92b848b3ba9cc4c4 P 6f1c68e27c0b4a0d96e01f5bf1931ddf reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f1c68e27c0b4a0d96e01f5bf1931ddf (127.23.167.195). New cstate: current_term: 1 leader_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } health_report { overall_health: HEALTHY } } }
I20260928 08:53:18.016474 26102 raft_consensus.cc:1261] T 233226cd3a29499c92b848b3ba9cc4c4 P 10d5981ef80e4fc5a634c389c7b512f0 [term 1 FOLLOWER]: Refusing update from remote peer 6f1c68e27c0b4a0d96e01f5bf1931ddf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:53:18.016479 26242 raft_consensus.cc:1261] T 233226cd3a29499c92b848b3ba9cc4c4 P 8891e45017924cd794b7860438bb8cc0 [term 1 FOLLOWER]: Refusing update from remote peer 6f1c68e27c0b4a0d96e01f5bf1931ddf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:53:18.016754 26459 consensus_queue.cc:1035] T 233226cd3a29499c92b848b3ba9cc4c4 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [LEADER]: Connected to new peer: Peer: permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:53:18.016917 26459 consensus_queue.cc:1035] T 233226cd3a29499c92b848b3ba9cc4c4 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:53:18.018122 26480 mvcc.cc:205] Tried to move back new op lower bound from 7334238609474285568 to 7334238609392484352. Current Snapshot: MvccSnapshot[applied={T|T < 7334238609474285568}]
W20260928 08:53:18.021538 26481 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 08:53:19.538268 25945 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:53:19.538364 25945 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:53:19.539130 25945 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e718379167694065b3c67180d47b82c0: Sending DeleteTablet for 3 replicas of tablet 233226cd3a29499c92b848b3ba9cc4c4
I20260928 08:53:19.539374 26082 tablet_service.cc:1514] Processing DeleteTablet for tablet 233226cd3a29499c92b848b3ba9cc4c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:53:19 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:34124
I20260928 08:53:19.539368 26222 tablet_service.cc:1514] Processing DeleteTablet for tablet 233226cd3a29499c92b848b3ba9cc4c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:53:19 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:42570
I20260928 08:53:19.539407 26346 tablet_service.cc:1514] Processing DeleteTablet for tablet 233226cd3a29499c92b848b3ba9cc4c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:53:19 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:48534
I20260928 08:53:19.539589 26529 tablet_replica.cc:330] T 233226cd3a29499c92b848b3ba9cc4c4 P 6f1c68e27c0b4a0d96e01f5bf1931ddf: stopping tablet replica
I20260928 08:53:19.539640 26529 raft_consensus.cc:2229] T 233226cd3a29499c92b848b3ba9cc4c4 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:53:19.539844 26529 raft_consensus.cc:2258] T 233226cd3a29499c92b848b3ba9cc4c4 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:53:19.539971 26528 tablet_replica.cc:330] T 233226cd3a29499c92b848b3ba9cc4c4 P 8891e45017924cd794b7860438bb8cc0: stopping tablet replica
I20260928 08:53:19.540027 26528 raft_consensus.cc:2229] T 233226cd3a29499c92b848b3ba9cc4c4 P 8891e45017924cd794b7860438bb8cc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:53:19.540064 26528 raft_consensus.cc:2258] T 233226cd3a29499c92b848b3ba9cc4c4 P 8891e45017924cd794b7860438bb8cc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:53:19.540058 26529 ts_tablet_manager.cc:1905] T 233226cd3a29499c92b848b3ba9cc4c4 P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:53:19.540189 26527 tablet_replica.cc:330] T 233226cd3a29499c92b848b3ba9cc4c4 P 10d5981ef80e4fc5a634c389c7b512f0: stopping tablet replica
I20260928 08:53:19.540350 26528 ts_tablet_manager.cc:1905] T 233226cd3a29499c92b848b3ba9cc4c4 P 8891e45017924cd794b7860438bb8cc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:53:19.541812 26529 ts_tablet_manager.cc:1918] T 233226cd3a29499c92b848b3ba9cc4c4 P 6f1c68e27c0b4a0d96e01f5bf1931ddf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 08:53:19.541882 26527 raft_consensus.cc:2229] T 233226cd3a29499c92b848b3ba9cc4c4 P 10d5981ef80e4fc5a634c389c7b512f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:53:19.541929 26529 log.cc:1196] T 233226cd3a29499c92b848b3ba9cc4c4 P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-2/wal/wals/233226cd3a29499c92b848b3ba9cc4c4
I20260928 08:53:19.542086 26528 ts_tablet_manager.cc:1918] T 233226cd3a29499c92b848b3ba9cc4c4 P 8891e45017924cd794b7860438bb8cc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 08:53:19.542145 26528 log.cc:1196] T 233226cd3a29499c92b848b3ba9cc4c4 P 8891e45017924cd794b7860438bb8cc0: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-1/wal/wals/233226cd3a29499c92b848b3ba9cc4c4
I20260928 08:53:19.542186 26529 ts_tablet_manager.cc:1939] T 233226cd3a29499c92b848b3ba9cc4c4 P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Deleting consensus metadata
I20260928 08:53:19.542351 26528 ts_tablet_manager.cc:1939] T 233226cd3a29499c92b848b3ba9cc4c4 P 8891e45017924cd794b7860438bb8cc0: Deleting consensus metadata
I20260928 08:53:19.542517 25931 catalog_manager.cc:4841] TS 6f1c68e27c0b4a0d96e01f5bf1931ddf (127.23.167.195:45937): tablet 233226cd3a29499c92b848b3ba9cc4c4 (table test-table [id=679a561a9a8148749b25018dce650a4c]) successfully deleted
I20260928 08:53:19.542593 26527 raft_consensus.cc:2258] T 233226cd3a29499c92b848b3ba9cc4c4 P 10d5981ef80e4fc5a634c389c7b512f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:53:19.542629 25931 catalog_manager.cc:4841] TS 8891e45017924cd794b7860438bb8cc0 (127.23.167.194:33449): tablet 233226cd3a29499c92b848b3ba9cc4c4 (table test-table [id=679a561a9a8148749b25018dce650a4c]) successfully deleted
I20260928 08:53:19.542820 26527 ts_tablet_manager.cc:1905] T 233226cd3a29499c92b848b3ba9cc4c4 P 10d5981ef80e4fc5a634c389c7b512f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:53:19.544250 26527 ts_tablet_manager.cc:1918] T 233226cd3a29499c92b848b3ba9cc4c4 P 10d5981ef80e4fc5a634c389c7b512f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 08:53:19.544309 26527 log.cc:1196] T 233226cd3a29499c92b848b3ba9cc4c4 P 10d5981ef80e4fc5a634c389c7b512f0: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-0/wal/wals/233226cd3a29499c92b848b3ba9cc4c4
I20260928 08:53:19.544513 26527 ts_tablet_manager.cc:1939] T 233226cd3a29499c92b848b3ba9cc4c4 P 10d5981ef80e4fc5a634c389c7b512f0: Deleting consensus metadata
I20260928 08:53:19.544885 25930 catalog_manager.cc:4841] TS 10d5981ef80e4fc5a634c389c7b512f0 (127.23.167.193:42119): tablet 233226cd3a29499c92b848b3ba9cc4c4 (table test-table [id=679a561a9a8148749b25018dce650a4c]) successfully deleted
Sep 28 08:53:19 dist-test-slave-pbjp krb5kdc[25900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585599, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 08:53:19 dist-test-slave-pbjp krb5kdc[25900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585599, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:19.561800 26537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:19.555349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39550 (local address 127.23.167.254:46595)
0928 08:53:19.555495 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:19.555496 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:19.555523 (+ 27us) server_negotiation.cc:407] Connection header received
0928 08:53:19.555545 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:19.555545 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:19.555556 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:19.555602 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:19.556086 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:19.556590 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:19.557358 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:19.557594 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:19.557637 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:19.559620 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:19.559621 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:19.559621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:19.559628 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:19.560922 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:19.561234 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:19.561235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:19.561235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:19.561287 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:19.561408 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:19.561409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:19.561409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:19.561508 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:19.561577 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:19.561651 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 08:53:19.561715 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
I20260928 08:53:19.563146 25945 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:53:19.563297 25945 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:53:19.564531 26082 tablet_service.cc:1467] Processing CreateTablet for tablet 9e570fc676c44c6298474e1b00224f5b (DEFAULT_TABLE table=test-table [id=a193ea5428b448ee963e45ed774e7338]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:53:19.564563 26222 tablet_service.cc:1467] Processing CreateTablet for tablet 9e570fc676c44c6298474e1b00224f5b (DEFAULT_TABLE table=test-table [id=a193ea5428b448ee963e45ed774e7338]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:53:19.564586 26346 tablet_service.cc:1467] Processing CreateTablet for tablet 9e570fc676c44c6298474e1b00224f5b (DEFAULT_TABLE table=test-table [id=a193ea5428b448ee963e45ed774e7338]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:53:19.564669 26082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e570fc676c44c6298474e1b00224f5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:19.564690 26346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e570fc676c44c6298474e1b00224f5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:19.564690 26222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e570fc676c44c6298474e1b00224f5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:19.565943 26538 tablet_bootstrap.cc:492] T 9e570fc676c44c6298474e1b00224f5b P 10d5981ef80e4fc5a634c389c7b512f0: Bootstrap starting.
I20260928 08:53:19.566203 26539 tablet_bootstrap.cc:492] T 9e570fc676c44c6298474e1b00224f5b P 8891e45017924cd794b7860438bb8cc0: Bootstrap starting.
I20260928 08:53:19.566390 26538 tablet_bootstrap.cc:654] T 9e570fc676c44c6298474e1b00224f5b P 10d5981ef80e4fc5a634c389c7b512f0: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:19.566571 26539 tablet_bootstrap.cc:654] T 9e570fc676c44c6298474e1b00224f5b P 8891e45017924cd794b7860438bb8cc0: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:19.566841 26540 tablet_bootstrap.cc:492] T 9e570fc676c44c6298474e1b00224f5b P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Bootstrap starting.
I20260928 08:53:19.566926 26538 tablet_bootstrap.cc:492] T 9e570fc676c44c6298474e1b00224f5b P 10d5981ef80e4fc5a634c389c7b512f0: No bootstrap required, opened a new log
I20260928 08:53:19.566980 26538 ts_tablet_manager.cc:1397] T 9e570fc676c44c6298474e1b00224f5b P 10d5981ef80e4fc5a634c389c7b512f0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 08:53:19.567085 26539 tablet_bootstrap.cc:492] T 9e570fc676c44c6298474e1b00224f5b P 8891e45017924cd794b7860438bb8cc0: No bootstrap required, opened a new log
I20260928 08:53:19.567131 26539 ts_tablet_manager.cc:1397] T 9e570fc676c44c6298474e1b00224f5b P 8891e45017924cd794b7860438bb8cc0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 08:53:19.567190 26540 tablet_bootstrap.cc:654] T 9e570fc676c44c6298474e1b00224f5b P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:19.567144 26538 raft_consensus.cc:348] T 9e570fc676c44c6298474e1b00224f5b P 10d5981ef80e4fc5a634c389c7b512f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:19.567242 26538 raft_consensus.cc:374] T 9e570fc676c44c6298474e1b00224f5b P 10d5981ef80e4fc5a634c389c7b512f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:19.567286 26538 raft_consensus.cc:729] T 9e570fc676c44c6298474e1b00224f5b P 10d5981ef80e4fc5a634c389c7b512f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10d5981ef80e4fc5a634c389c7b512f0, State: Initialized, Role: FOLLOWER
I20260928 08:53:19.567286 26539 raft_consensus.cc:348] T 9e570fc676c44c6298474e1b00224f5b P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:19.567379 26539 raft_consensus.cc:374] T 9e570fc676c44c6298474e1b00224f5b P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:19.567358 26538 consensus_queue.cc:260] T 9e570fc676c44c6298474e1b00224f5b P 10d5981ef80e4fc5a634c389c7b512f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:19.567433 26539 raft_consensus.cc:729] T 9e570fc676c44c6298474e1b00224f5b P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8891e45017924cd794b7860438bb8cc0, State: Initialized, Role: FOLLOWER
I20260928 08:53:19.567492 26539 consensus_queue.cc:260] T 9e570fc676c44c6298474e1b00224f5b P 8891e45017924cd794b7860438bb8cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:19.567550 26538 ts_tablet_manager.cc:1428] T 9e570fc676c44c6298474e1b00224f5b P 10d5981ef80e4fc5a634c389c7b512f0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 08:53:19.567610 26539 ts_tablet_manager.cc:1428] T 9e570fc676c44c6298474e1b00224f5b P 8891e45017924cd794b7860438bb8cc0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:19.567754 26540 tablet_bootstrap.cc:492] T 9e570fc676c44c6298474e1b00224f5b P 6f1c68e27c0b4a0d96e01f5bf1931ddf: No bootstrap required, opened a new log
I20260928 08:53:19.567802 26540 ts_tablet_manager.cc:1397] T 9e570fc676c44c6298474e1b00224f5b P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 08:53:19.567945 26540 raft_consensus.cc:348] T 9e570fc676c44c6298474e1b00224f5b P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:19.568010 26540 raft_consensus.cc:374] T 9e570fc676c44c6298474e1b00224f5b P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:19.568027 26540 raft_consensus.cc:729] T 9e570fc676c44c6298474e1b00224f5b P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f1c68e27c0b4a0d96e01f5bf1931ddf, State: Initialized, Role: FOLLOWER
I20260928 08:53:19.568068 26540 consensus_queue.cc:260] T 9e570fc676c44c6298474e1b00224f5b P 6f1c68e27c0b4a0d96e01f5bf1931ddf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:19.568153 26540 ts_tablet_manager.cc:1428] T 9e570fc676c44c6298474e1b00224f5b P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:19.651985 26509 raft_consensus.cc:482] T 9e570fc676c44c6298474e1b00224f5b P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:53:19.652086 26509 raft_consensus.cc:504] T 9e570fc676c44c6298474e1b00224f5b P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:19.652254 26509 leader_election.cc:288] T 9e570fc676c44c6298474e1b00224f5b P 8891e45017924cd794b7860438bb8cc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10d5981ef80e4fc5a634c389c7b512f0 (127.23.167.193:42119), 6f1c68e27c0b4a0d96e01f5bf1931ddf (127.23.167.195:45937)
I20260928 08:53:19.652495 26382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e570fc676c44c6298474e1b00224f5b" candidate_uuid: "8891e45017924cd794b7860438bb8cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" is_pre_election: true
I20260928 08:53:19.652520 26101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e570fc676c44c6298474e1b00224f5b" candidate_uuid: "8891e45017924cd794b7860438bb8cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10d5981ef80e4fc5a634c389c7b512f0" is_pre_election: true
I20260928 08:53:19.652601 26101 raft_consensus.cc:2454] T 9e570fc676c44c6298474e1b00224f5b P 10d5981ef80e4fc5a634c389c7b512f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8891e45017924cd794b7860438bb8cc0 in term 0.
I20260928 08:53:19.652596 26382 raft_consensus.cc:2454] T 9e570fc676c44c6298474e1b00224f5b P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8891e45017924cd794b7860438bb8cc0 in term 0.
I20260928 08:53:19.652760 26175 leader_election.cc:302] T 9e570fc676c44c6298474e1b00224f5b P 8891e45017924cd794b7860438bb8cc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f1c68e27c0b4a0d96e01f5bf1931ddf, 8891e45017924cd794b7860438bb8cc0; no voters:
I20260928 08:53:19.652872 26509 raft_consensus.cc:2781] T 9e570fc676c44c6298474e1b00224f5b P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:53:19.652920 26509 raft_consensus.cc:482] T 9e570fc676c44c6298474e1b00224f5b P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:53:19.652940 26509 raft_consensus.cc:3037] T 9e570fc676c44c6298474e1b00224f5b P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:19.653699 26509 raft_consensus.cc:504] T 9e570fc676c44c6298474e1b00224f5b P 8891e45017924cd794b7860438bb8cc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:19.653828 26509 leader_election.cc:288] T 9e570fc676c44c6298474e1b00224f5b P 8891e45017924cd794b7860438bb8cc0 [CANDIDATE]: Term 1 election: Requested vote from peers 10d5981ef80e4fc5a634c389c7b512f0 (127.23.167.193:42119), 6f1c68e27c0b4a0d96e01f5bf1931ddf (127.23.167.195:45937)
I20260928 08:53:19.654006 26382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e570fc676c44c6298474e1b00224f5b" candidate_uuid: "8891e45017924cd794b7860438bb8cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf"
I20260928 08:53:19.654072 26382 raft_consensus.cc:3037] T 9e570fc676c44c6298474e1b00224f5b P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:19.654023 26101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e570fc676c44c6298474e1b00224f5b" candidate_uuid: "8891e45017924cd794b7860438bb8cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10d5981ef80e4fc5a634c389c7b512f0"
I20260928 08:53:19.654106 26101 raft_consensus.cc:3037] T 9e570fc676c44c6298474e1b00224f5b P 10d5981ef80e4fc5a634c389c7b512f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:19.654688 26382 raft_consensus.cc:2454] T 9e570fc676c44c6298474e1b00224f5b P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8891e45017924cd794b7860438bb8cc0 in term 1.
I20260928 08:53:19.654848 26175 leader_election.cc:302] T 9e570fc676c44c6298474e1b00224f5b P 8891e45017924cd794b7860438bb8cc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f1c68e27c0b4a0d96e01f5bf1931ddf, 8891e45017924cd794b7860438bb8cc0; no voters:
I20260928 08:53:19.654946 26509 raft_consensus.cc:2781] T 9e570fc676c44c6298474e1b00224f5b P 8891e45017924cd794b7860438bb8cc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:53:19.654995 26509 raft_consensus.cc:686] T 9e570fc676c44c6298474e1b00224f5b P 8891e45017924cd794b7860438bb8cc0 [term 1 LEADER]: Becoming Leader. State: Replica: 8891e45017924cd794b7860438bb8cc0, State: Running, Role: LEADER
I20260928 08:53:19.655045 26509 consensus_queue.cc:237] T 9e570fc676c44c6298474e1b00224f5b P 8891e45017924cd794b7860438bb8cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:19.655066 26101 raft_consensus.cc:2454] T 9e570fc676c44c6298474e1b00224f5b P 10d5981ef80e4fc5a634c389c7b512f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8891e45017924cd794b7860438bb8cc0 in term 1.
I20260928 08:53:19.655673 25944 catalog_manager.cc:5495] T 9e570fc676c44c6298474e1b00224f5b P 8891e45017924cd794b7860438bb8cc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8891e45017924cd794b7860438bb8cc0 (127.23.167.194). New cstate: current_term: 1 leader_uuid: "8891e45017924cd794b7860438bb8cc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } health_report { overall_health: UNKNOWN } } }
I20260928 08:53:19.665823 26547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:19.662387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42582 (local address 127.23.167.194:33449)
0928 08:53:19.662686 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:19.662688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:19.662696 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:53:19.662718 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:19.662719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:19.662731 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:19.662827 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:53:19.662988 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:19.663644 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:19.664773 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:19.665113 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:19.665179 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:19.665420 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:53:19.665477 (+ 57us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:53:19.665529 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:19.665606 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 08:53:19.665634 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":114,"threads_started":1}
I20260928 08:53:19.671672 26547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:19.668694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:47331 (local address 127.23.167.194:33449)
0928 08:53:19.668758 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:19.668760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:19.668813 (+ 53us) server_negotiation.cc:407] Connection header received
0928 08:53:19.668873 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:19.668874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:19.668883 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:19.668959 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:53:19.669124 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:19.669666 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:19.670823 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:19.671121 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:19.671195 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:19.671405 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:53:19.671441 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:53:19.671494 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:19.671542 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 08:53:19.671572 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":21}
I20260928 08:53:19.671694 26550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:19.668613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:33449 (local address 127.23.167.194:47331)
0928 08:53:19.668741 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:19.668751 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:19.668826 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:19.668965 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:19.668966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:19.668967 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:53:19.669030 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:19.669032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:19.669872 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:19.669873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:19.670714 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:19.670719 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:19.671259 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:19.671260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:19.671329 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:19.671335 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:53:19.671498 (+ 163us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:53:19.671499 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 08:53:19.671539 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 08:53:19.671592 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":4,"thread_start_us":71,"threads_started":1}
I20260928 08:53:19.673643 26382 raft_consensus.cc:1261] T 9e570fc676c44c6298474e1b00224f5b P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 1 FOLLOWER]: Refusing update from remote peer 8891e45017924cd794b7860438bb8cc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:53:19.673743 26102 raft_consensus.cc:1261] T 9e570fc676c44c6298474e1b00224f5b P 10d5981ef80e4fc5a634c389c7b512f0 [term 1 FOLLOWER]: Refusing update from remote peer 8891e45017924cd794b7860438bb8cc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:53:19.673848 26514 consensus_queue.cc:1035] T 9e570fc676c44c6298474e1b00224f5b P 8891e45017924cd794b7860438bb8cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:53:19.674083 26514 consensus_queue.cc:1035] T 9e570fc676c44c6298474e1b00224f5b P 8891e45017924cd794b7860438bb8cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:53:19.688126 25944 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:53:19.688239 25944 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:53:19.688871 25944 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e718379167694065b3c67180d47b82c0: Sending DeleteTablet for 3 replicas of tablet 9e570fc676c44c6298474e1b00224f5b
I20260928 08:53:19.689195 26082 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e570fc676c44c6298474e1b00224f5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:53:19 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:34124
I20260928 08:53:19.689244 26346 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e570fc676c44c6298474e1b00224f5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:53:19 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:48534
I20260928 08:53:19.689334 26222 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e570fc676c44c6298474e1b00224f5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:53:19 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:42570
I20260928 08:53:19.689414 26527 tablet_replica.cc:330] T 9e570fc676c44c6298474e1b00224f5b P 10d5981ef80e4fc5a634c389c7b512f0: stopping tablet replica
I20260928 08:53:19.689431 26528 tablet_replica.cc:330] T 9e570fc676c44c6298474e1b00224f5b P 8891e45017924cd794b7860438bb8cc0: stopping tablet replica
I20260928 08:53:19.689472 26527 raft_consensus.cc:2229] T 9e570fc676c44c6298474e1b00224f5b P 10d5981ef80e4fc5a634c389c7b512f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:53:19.689502 26528 raft_consensus.cc:2229] T 9e570fc676c44c6298474e1b00224f5b P 8891e45017924cd794b7860438bb8cc0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:53:19.689512 26527 raft_consensus.cc:2258] T 9e570fc676c44c6298474e1b00224f5b P 10d5981ef80e4fc5a634c389c7b512f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:53:19.689575 26529 tablet_replica.cc:330] T 9e570fc676c44c6298474e1b00224f5b P 6f1c68e27c0b4a0d96e01f5bf1931ddf: stopping tablet replica
I20260928 08:53:19.689591 26528 raft_consensus.cc:2258] T 9e570fc676c44c6298474e1b00224f5b P 8891e45017924cd794b7860438bb8cc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:53:19.689625 26529 raft_consensus.cc:2229] T 9e570fc676c44c6298474e1b00224f5b P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:53:19.689656 26529 raft_consensus.cc:2258] T 9e570fc676c44c6298474e1b00224f5b P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:53:19.689805 25944 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:53:19.689942 25944 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:53:19.690275 26528 ts_tablet_manager.cc:1905] T 9e570fc676c44c6298474e1b00224f5b P 8891e45017924cd794b7860438bb8cc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:53:19.690290 26527 ts_tablet_manager.cc:1905] T 9e570fc676c44c6298474e1b00224f5b P 10d5981ef80e4fc5a634c389c7b512f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:53:19.691183 26346 tablet_service.cc:1467] Processing CreateTablet for tablet fb35aa98aa834f158afe1554db1da4a9 (DEFAULT_TABLE table=test-table [id=2baba1d102ee45f787efad5137259a54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:53:19.691286 26346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb35aa98aa834f158afe1554db1da4a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:19.691711 26082 tablet_service.cc:1467] Processing CreateTablet for tablet fb35aa98aa834f158afe1554db1da4a9 (DEFAULT_TABLE table=test-table [id=2baba1d102ee45f787efad5137259a54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:53:19.692045 26528 ts_tablet_manager.cc:1918] T 9e570fc676c44c6298474e1b00224f5b P 8891e45017924cd794b7860438bb8cc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:53:19.692050 26527 ts_tablet_manager.cc:1918] T 9e570fc676c44c6298474e1b00224f5b P 10d5981ef80e4fc5a634c389c7b512f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:53:19.692099 26527 log.cc:1196] T 9e570fc676c44c6298474e1b00224f5b P 10d5981ef80e4fc5a634c389c7b512f0: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-0/wal/wals/9e570fc676c44c6298474e1b00224f5b
I20260928 08:53:19.692102 26528 log.cc:1196] T 9e570fc676c44c6298474e1b00224f5b P 8891e45017924cd794b7860438bb8cc0: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-1/wal/wals/9e570fc676c44c6298474e1b00224f5b
I20260928 08:53:19.692289 26527 ts_tablet_manager.cc:1939] T 9e570fc676c44c6298474e1b00224f5b P 10d5981ef80e4fc5a634c389c7b512f0: Deleting consensus metadata
I20260928 08:53:19.692327 26528 ts_tablet_manager.cc:1939] T 9e570fc676c44c6298474e1b00224f5b P 8891e45017924cd794b7860438bb8cc0: Deleting consensus metadata
I20260928 08:53:19.692435 26082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb35aa98aa834f158afe1554db1da4a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:19.692554 26540 tablet_bootstrap.cc:492] T fb35aa98aa834f158afe1554db1da4a9 P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Bootstrap starting.
I20260928 08:53:19.692653 25931 catalog_manager.cc:4841] TS 8891e45017924cd794b7860438bb8cc0 (127.23.167.194:33449): tablet 9e570fc676c44c6298474e1b00224f5b (table test-table [id=a193ea5428b448ee963e45ed774e7338]) successfully deleted
I20260928 08:53:19.692775 26222 tablet_service.cc:1467] Processing CreateTablet for tablet fb35aa98aa834f158afe1554db1da4a9 (DEFAULT_TABLE table=test-table [id=2baba1d102ee45f787efad5137259a54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:53:19.692859 26222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb35aa98aa834f158afe1554db1da4a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:19.692871 26540 tablet_bootstrap.cc:654] T fb35aa98aa834f158afe1554db1da4a9 P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:19.693517 26538 tablet_bootstrap.cc:492] T fb35aa98aa834f158afe1554db1da4a9 P 10d5981ef80e4fc5a634c389c7b512f0: Bootstrap starting.
I20260928 08:53:19.693598 26540 tablet_bootstrap.cc:492] T fb35aa98aa834f158afe1554db1da4a9 P 6f1c68e27c0b4a0d96e01f5bf1931ddf: No bootstrap required, opened a new log
I20260928 08:53:19.693637 26540 ts_tablet_manager.cc:1397] T fb35aa98aa834f158afe1554db1da4a9 P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 08:53:19.693734 26540 raft_consensus.cc:348] T fb35aa98aa834f158afe1554db1da4a9 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:19.693790 26540 raft_consensus.cc:374] T fb35aa98aa834f158afe1554db1da4a9 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:19.693809 26540 raft_consensus.cc:729] T fb35aa98aa834f158afe1554db1da4a9 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f1c68e27c0b4a0d96e01f5bf1931ddf, State: Initialized, Role: FOLLOWER
I20260928 08:53:19.693864 26540 consensus_queue.cc:260] T fb35aa98aa834f158afe1554db1da4a9 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:19.693926 26538 tablet_bootstrap.cc:654] T fb35aa98aa834f158afe1554db1da4a9 P 10d5981ef80e4fc5a634c389c7b512f0: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:19.693953 26540 ts_tablet_manager.cc:1428] T fb35aa98aa834f158afe1554db1da4a9 P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:19.694135 25930 catalog_manager.cc:4841] TS 10d5981ef80e4fc5a634c389c7b512f0 (127.23.167.193:42119): tablet 9e570fc676c44c6298474e1b00224f5b (table test-table [id=a193ea5428b448ee963e45ed774e7338]) successfully deleted
I20260928 08:53:19.694250 26539 tablet_bootstrap.cc:492] T fb35aa98aa834f158afe1554db1da4a9 P 8891e45017924cd794b7860438bb8cc0: Bootstrap starting.
I20260928 08:53:19.694589 26539 tablet_bootstrap.cc:654] T fb35aa98aa834f158afe1554db1da4a9 P 8891e45017924cd794b7860438bb8cc0: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:19.695065 26529 ts_tablet_manager.cc:1905] T 9e570fc676c44c6298474e1b00224f5b P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:53:19.695101 26538 tablet_bootstrap.cc:492] T fb35aa98aa834f158afe1554db1da4a9 P 10d5981ef80e4fc5a634c389c7b512f0: No bootstrap required, opened a new log
I20260928 08:53:19.695134 26538 ts_tablet_manager.cc:1397] T fb35aa98aa834f158afe1554db1da4a9 P 10d5981ef80e4fc5a634c389c7b512f0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:53:19.695236 26538 raft_consensus.cc:348] T fb35aa98aa834f158afe1554db1da4a9 P 10d5981ef80e4fc5a634c389c7b512f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:19.695278 26538 raft_consensus.cc:374] T fb35aa98aa834f158afe1554db1da4a9 P 10d5981ef80e4fc5a634c389c7b512f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:19.695288 26538 raft_consensus.cc:729] T fb35aa98aa834f158afe1554db1da4a9 P 10d5981ef80e4fc5a634c389c7b512f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10d5981ef80e4fc5a634c389c7b512f0, State: Initialized, Role: FOLLOWER
I20260928 08:53:19.695322 26538 consensus_queue.cc:260] T fb35aa98aa834f158afe1554db1da4a9 P 10d5981ef80e4fc5a634c389c7b512f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:19.695399 26538 ts_tablet_manager.cc:1428] T fb35aa98aa834f158afe1554db1da4a9 P 10d5981ef80e4fc5a634c389c7b512f0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:19.695715 26539 tablet_bootstrap.cc:492] T fb35aa98aa834f158afe1554db1da4a9 P 8891e45017924cd794b7860438bb8cc0: No bootstrap required, opened a new log
I20260928 08:53:19.695755 26539 ts_tablet_manager.cc:1397] T fb35aa98aa834f158afe1554db1da4a9 P 8891e45017924cd794b7860438bb8cc0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:53:19.695860 26539 raft_consensus.cc:348] T fb35aa98aa834f158afe1554db1da4a9 P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:19.695919 26539 raft_consensus.cc:374] T fb35aa98aa834f158afe1554db1da4a9 P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:19.695940 26539 raft_consensus.cc:729] T fb35aa98aa834f158afe1554db1da4a9 P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8891e45017924cd794b7860438bb8cc0, State: Initialized, Role: FOLLOWER
I20260928 08:53:19.695979 26539 consensus_queue.cc:260] T fb35aa98aa834f158afe1554db1da4a9 P 8891e45017924cd794b7860438bb8cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:19.696062 26539 ts_tablet_manager.cc:1428] T fb35aa98aa834f158afe1554db1da4a9 P 8891e45017924cd794b7860438bb8cc0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:19.696544 26529 ts_tablet_manager.cc:1918] T 9e570fc676c44c6298474e1b00224f5b P 6f1c68e27c0b4a0d96e01f5bf1931ddf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:53:19.696602 26529 log.cc:1196] T 9e570fc676c44c6298474e1b00224f5b P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-2/wal/wals/9e570fc676c44c6298474e1b00224f5b
I20260928 08:53:19.696748 26529 ts_tablet_manager.cc:1939] T 9e570fc676c44c6298474e1b00224f5b P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Deleting consensus metadata
I20260928 08:53:19.697006 25931 catalog_manager.cc:4841] TS 6f1c68e27c0b4a0d96e01f5bf1931ddf (127.23.167.195:45937): tablet 9e570fc676c44c6298474e1b00224f5b (table test-table [id=a193ea5428b448ee963e45ed774e7338]) successfully deleted
I20260928 08:53:19.713497 26514 raft_consensus.cc:482] T fb35aa98aa834f158afe1554db1da4a9 P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:53:19.713610 26514 raft_consensus.cc:504] T fb35aa98aa834f158afe1554db1da4a9 P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:19.713814 26514 leader_election.cc:288] T fb35aa98aa834f158afe1554db1da4a9 P 8891e45017924cd794b7860438bb8cc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10d5981ef80e4fc5a634c389c7b512f0 (127.23.167.193:42119), 6f1c68e27c0b4a0d96e01f5bf1931ddf (127.23.167.195:45937)
I20260928 08:53:19.713970 26101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb35aa98aa834f158afe1554db1da4a9" candidate_uuid: "8891e45017924cd794b7860438bb8cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10d5981ef80e4fc5a634c389c7b512f0" is_pre_election: true
I20260928 08:53:19.714071 26101 raft_consensus.cc:2454] T fb35aa98aa834f158afe1554db1da4a9 P 10d5981ef80e4fc5a634c389c7b512f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8891e45017924cd794b7860438bb8cc0 in term 0.
I20260928 08:53:19.714038 26382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb35aa98aa834f158afe1554db1da4a9" candidate_uuid: "8891e45017924cd794b7860438bb8cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" is_pre_election: true
I20260928 08:53:19.714128 26382 raft_consensus.cc:2454] T fb35aa98aa834f158afe1554db1da4a9 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8891e45017924cd794b7860438bb8cc0 in term 0.
I20260928 08:53:19.714262 26175 leader_election.cc:302] T fb35aa98aa834f158afe1554db1da4a9 P 8891e45017924cd794b7860438bb8cc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f1c68e27c0b4a0d96e01f5bf1931ddf, 8891e45017924cd794b7860438bb8cc0; no voters:
I20260928 08:53:19.714358 26514 raft_consensus.cc:2781] T fb35aa98aa834f158afe1554db1da4a9 P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:53:19.714395 26514 raft_consensus.cc:482] T fb35aa98aa834f158afe1554db1da4a9 P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:53:19.714421 26514 raft_consensus.cc:3037] T fb35aa98aa834f158afe1554db1da4a9 P 8891e45017924cd794b7860438bb8cc0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:19.715294 26514 raft_consensus.cc:504] T fb35aa98aa834f158afe1554db1da4a9 P 8891e45017924cd794b7860438bb8cc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:19.715440 26514 leader_election.cc:288] T fb35aa98aa834f158afe1554db1da4a9 P 8891e45017924cd794b7860438bb8cc0 [CANDIDATE]: Term 1 election: Requested vote from peers 10d5981ef80e4fc5a634c389c7b512f0 (127.23.167.193:42119), 6f1c68e27c0b4a0d96e01f5bf1931ddf (127.23.167.195:45937)
I20260928 08:53:19.715593 26101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb35aa98aa834f158afe1554db1da4a9" candidate_uuid: "8891e45017924cd794b7860438bb8cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10d5981ef80e4fc5a634c389c7b512f0"
I20260928 08:53:19.715620 26382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb35aa98aa834f158afe1554db1da4a9" candidate_uuid: "8891e45017924cd794b7860438bb8cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf"
I20260928 08:53:19.715663 26101 raft_consensus.cc:3037] T fb35aa98aa834f158afe1554db1da4a9 P 10d5981ef80e4fc5a634c389c7b512f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:19.715693 26382 raft_consensus.cc:3037] T fb35aa98aa834f158afe1554db1da4a9 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:19.716426 26101 raft_consensus.cc:2454] T fb35aa98aa834f158afe1554db1da4a9 P 10d5981ef80e4fc5a634c389c7b512f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8891e45017924cd794b7860438bb8cc0 in term 1.
I20260928 08:53:19.716493 26382 raft_consensus.cc:2454] T fb35aa98aa834f158afe1554db1da4a9 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8891e45017924cd794b7860438bb8cc0 in term 1.
I20260928 08:53:19.716660 26175 leader_election.cc:302] T fb35aa98aa834f158afe1554db1da4a9 P 8891e45017924cd794b7860438bb8cc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f1c68e27c0b4a0d96e01f5bf1931ddf, 8891e45017924cd794b7860438bb8cc0; no voters:
I20260928 08:53:19.716745 26514 raft_consensus.cc:2781] T fb35aa98aa834f158afe1554db1da4a9 P 8891e45017924cd794b7860438bb8cc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:53:19.716785 26514 raft_consensus.cc:686] T fb35aa98aa834f158afe1554db1da4a9 P 8891e45017924cd794b7860438bb8cc0 [term 1 LEADER]: Becoming Leader. State: Replica: 8891e45017924cd794b7860438bb8cc0, State: Running, Role: LEADER
I20260928 08:53:19.716845 26514 consensus_queue.cc:237] T fb35aa98aa834f158afe1554db1da4a9 P 8891e45017924cd794b7860438bb8cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } }
I20260928 08:53:19.717482 25944 catalog_manager.cc:5495] T fb35aa98aa834f158afe1554db1da4a9 P 8891e45017924cd794b7860438bb8cc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8891e45017924cd794b7860438bb8cc0 (127.23.167.194). New cstate: current_term: 1 leader_uuid: "8891e45017924cd794b7860438bb8cc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8891e45017924cd794b7860438bb8cc0" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 } health_report { overall_health: UNKNOWN } } }
I20260928 08:53:19.728852 26382 raft_consensus.cc:1261] T fb35aa98aa834f158afe1554db1da4a9 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 1 FOLLOWER]: Refusing update from remote peer 8891e45017924cd794b7860438bb8cc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:53:19.728916 26101 raft_consensus.cc:1261] T fb35aa98aa834f158afe1554db1da4a9 P 10d5981ef80e4fc5a634c389c7b512f0 [term 1 FOLLOWER]: Refusing update from remote peer 8891e45017924cd794b7860438bb8cc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:53:19.729049 26509 consensus_queue.cc:1035] T fb35aa98aa834f158afe1554db1da4a9 P 8891e45017924cd794b7860438bb8cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f1c68e27c0b4a0d96e01f5bf1931ddf" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:53:19.729189 26514 consensus_queue.cc:1035] T fb35aa98aa834f158afe1554db1da4a9 P 8891e45017924cd794b7860438bb8cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10d5981ef80e4fc5a634c389c7b512f0" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 42119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 08:53:19.733508 26507 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 08:53:21.252763 25945 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:53:21.252875 25945 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:53:21.253711 25945 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e718379167694065b3c67180d47b82c0: Sending DeleteTablet for 3 replicas of tablet fb35aa98aa834f158afe1554db1da4a9
I20260928 08:53:21.253944 26082 tablet_service.cc:1514] Processing DeleteTablet for tablet fb35aa98aa834f158afe1554db1da4a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:53:21 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:34124
I20260928 08:53:21.253973 26346 tablet_service.cc:1514] Processing DeleteTablet for tablet fb35aa98aa834f158afe1554db1da4a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:53:21 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:48534
I20260928 08:53:21.254130 26222 tablet_service.cc:1514] Processing DeleteTablet for tablet fb35aa98aa834f158afe1554db1da4a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:53:21 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:42570
I20260928 08:53:21.254144 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 26017
I20260928 08:53:21.255445 26580 tablet_replica.cc:330] T fb35aa98aa834f158afe1554db1da4a9 P 6f1c68e27c0b4a0d96e01f5bf1931ddf: stopping tablet replica
I20260928 08:53:21.255517 26580 raft_consensus.cc:2229] T fb35aa98aa834f158afe1554db1da4a9 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:53:21.255559 26580 raft_consensus.cc:2258] T fb35aa98aa834f158afe1554db1da4a9 P 6f1c68e27c0b4a0d96e01f5bf1931ddf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:53:21.255679 26582 tablet_replica.cc:330] T fb35aa98aa834f158afe1554db1da4a9 P 8891e45017924cd794b7860438bb8cc0: stopping tablet replica
I20260928 08:53:21.255731 26582 raft_consensus.cc:2229] T fb35aa98aa834f158afe1554db1da4a9 P 8891e45017924cd794b7860438bb8cc0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:53:21.255787 26582 raft_consensus.cc:2258] T fb35aa98aa834f158afe1554db1da4a9 P 8891e45017924cd794b7860438bb8cc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:53:21.255800 26580 ts_tablet_manager.cc:1905] T fb35aa98aa834f158afe1554db1da4a9 P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:53:21.256016 26582 ts_tablet_manager.cc:1905] T fb35aa98aa834f158afe1554db1da4a9 P 8891e45017924cd794b7860438bb8cc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:53:21.257406 26580 ts_tablet_manager.cc:1918] T fb35aa98aa834f158afe1554db1da4a9 P 6f1c68e27c0b4a0d96e01f5bf1931ddf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 08:53:21.257460 26580 log.cc:1196] T fb35aa98aa834f158afe1554db1da4a9 P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-2/wal/wals/fb35aa98aa834f158afe1554db1da4a9
I20260928 08:53:21.257645 26580 ts_tablet_manager.cc:1939] T fb35aa98aa834f158afe1554db1da4a9 P 6f1c68e27c0b4a0d96e01f5bf1931ddf: Deleting consensus metadata
I20260928 08:53:21.257678 26582 ts_tablet_manager.cc:1918] T fb35aa98aa834f158afe1554db1da4a9 P 8891e45017924cd794b7860438bb8cc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 08:53:21.257735 26582 log.cc:1196] T fb35aa98aa834f158afe1554db1da4a9 P 8891e45017924cd794b7860438bb8cc0: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585589999956-24223-0/minicluster-data/ts-1/wal/wals/fb35aa98aa834f158afe1554db1da4a9
I20260928 08:53:21.257951 26582 ts_tablet_manager.cc:1939] T fb35aa98aa834f158afe1554db1da4a9 P 8891e45017924cd794b7860438bb8cc0: Deleting consensus metadata
I20260928 08:53:21.257967 25931 catalog_manager.cc:4841] TS 6f1c68e27c0b4a0d96e01f5bf1931ddf (127.23.167.195:45937): tablet fb35aa98aa834f158afe1554db1da4a9 (table test-table [id=2baba1d102ee45f787efad5137259a54]) successfully deleted
I20260928 08:53:21.258417 25931 catalog_manager.cc:4841] TS 8891e45017924cd794b7860438bb8cc0 (127.23.167.194:33449): tablet fb35aa98aa834f158afe1554db1da4a9 (table test-table [id=2baba1d102ee45f787efad5137259a54]) successfully deleted
W20260928 08:53:21.260311 25930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.167.193:42119 (error 108)
I20260928 08:53:21.260514 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 26157
I20260928 08:53:21.261803 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:21.260638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:42119 (local address 127.0.0.1:34126)
0928 08:53:21.261725 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:21.261770 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.193:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1064,"thread_start_us":53,"threads_started":1}
W20260928 08:53:21.261932 25930 catalog_manager.cc:4584] TS 10d5981ef80e4fc5a634c389c7b512f0 (127.23.167.193:42119): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fb35aa98aa834f158afe1554db1da4a9: Network error: Client connection negotiation failed: client connection to 127.23.167.193:42119: connect: Connection refused (error 111)
I20260928 08:53:21.265456 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 26297
I20260928 08:53:21.270109 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 25913
2026-09-28T08:53:21Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5587 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:53:21 dist-test-slave-pbjp krb5kdc[26587](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:53:21 dist-test-slave-pbjp krb5kdc[26587](info): set up 2 sockets
Sep 28 08:53:21 dist-test-slave-pbjp krb5kdc[26587](info): commencing operation
krb5kdc: starting...
W20260928 08:53:22.228523 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:53:22 dist-test-slave-pbjp krb5kdc[26587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:53:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:53:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.167.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
I20260928 08:53:22.340453 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:38151
--webserver_interface=127.23.167.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:38151
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:22.420457 26600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:22.420681 26600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:22.420703 26600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:22.421908 26600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:53:22.421949 26600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:22.421962 26600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:22.421973 26600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:53:22.421984 26600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:53:22.423458 26600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:38151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:38151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:22.423666 26600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:22.423851 26600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:22.428755 26608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:22.428746 26605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:22.428740 26606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:22.428969 26600 server_base.cc:1034] running on GCE node
I20260928 08:53:22.429308 26600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:22.429666 26600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:22.430832 26600 hybrid_clock.cc:648] HybridClock initialized: now 1790585602430822 us; error 34 us; skew 500 ppm
Sep 28 08:53:22 dist-test-slave-pbjp krb5kdc[26587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585602, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:22.433320 26600 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:53:22.433825 26600 webserver.cc:460] Webserver started at http://127.23.167.254:40645/ using document root <none> and password file <none>
I20260928 08:53:22.434047 26600 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:22.434098 26600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:22.434218 26600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:22.435281 26600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "f0e46a3c1e7546d8b03a472ef7c0d5b5"
format_stamp: "Formatted at 2026-09-28 08:53:22 on dist-test-slave-pbjp"
server_key: "d9d9b88d0b6b100bf8924c265f47bb8a"
server_key_iv: "6ca75cbe91559ae4729855ebc7c90d65"
server_key_version: "encryptionkey@0"
I20260928 08:53:22.435624 26600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "f0e46a3c1e7546d8b03a472ef7c0d5b5"
format_stamp: "Formatted at 2026-09-28 08:53:22 on dist-test-slave-pbjp"
server_key: "d9d9b88d0b6b100bf8924c265f47bb8a"
server_key_iv: "6ca75cbe91559ae4729855ebc7c90d65"
server_key_version: "encryptionkey@0"
I20260928 08:53:22.436884 26600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:53:22.437649 26615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:22.437798 26600 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:22.437882 26600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "f0e46a3c1e7546d8b03a472ef7c0d5b5"
format_stamp: "Formatted at 2026-09-28 08:53:22 on dist-test-slave-pbjp"
server_key: "d9d9b88d0b6b100bf8924c265f47bb8a"
server_key_iv: "6ca75cbe91559ae4729855ebc7c90d65"
server_key_version: "encryptionkey@0"
I20260928 08:53:22.437950 26600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:22.464314 26600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:22.466521 26600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:22.466665 26600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:22.470374 26600 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:38151
I20260928 08:53:22.470414 26667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:38151 every 8 connection(s)
I20260928 08:53:22.470749 26600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:53:22.471599 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:22.473958 26668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0e46a3c1e7546d8b03a472ef7c0d5b5: Bootstrap starting.
I20260928 08:53:22.474509 26668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0e46a3c1e7546d8b03a472ef7c0d5b5: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:22.474771 26668 log.cc:824] T 00000000000000000000000000000000 P f0e46a3c1e7546d8b03a472ef7c0d5b5: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:22.475275 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 26600
I20260928 08:53:22.475345 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:53:22.475412 26668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0e46a3c1e7546d8b03a472ef7c0d5b5: No bootstrap required, opened a new log
I20260928 08:53:22.475497 24223 external_mini_cluster.cc:1428] Setting key f3f392a721413a21d2b8660c756d91a0
I20260928 08:53:22.477396 26668 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0e46a3c1e7546d8b03a472ef7c0d5b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e46a3c1e7546d8b03a472ef7c0d5b5" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38151 } }
I20260928 08:53:22.477576 26668 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0e46a3c1e7546d8b03a472ef7c0d5b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:22.477607 26668 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0e46a3c1e7546d8b03a472ef7c0d5b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e46a3c1e7546d8b03a472ef7c0d5b5, State: Initialized, Role: FOLLOWER
I20260928 08:53:22.477741 26668 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0e46a3c1e7546d8b03a472ef7c0d5b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e46a3c1e7546d8b03a472ef7c0d5b5" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38151 } }
I20260928 08:53:22.477828 26668 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0e46a3c1e7546d8b03a472ef7c0d5b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:53:22.477865 26668 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0e46a3c1e7546d8b03a472ef7c0d5b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:53:22.477909 26668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0e46a3c1e7546d8b03a472ef7c0d5b5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:22.478554 26668 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0e46a3c1e7546d8b03a472ef7c0d5b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e46a3c1e7546d8b03a472ef7c0d5b5" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38151 } }
I20260928 08:53:22.478657 26668 leader_election.cc:302] T 00000000000000000000000000000000 P f0e46a3c1e7546d8b03a472ef7c0d5b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0e46a3c1e7546d8b03a472ef7c0d5b5; no voters:
I20260928 08:53:22.478798 26668 leader_election.cc:288] T 00000000000000000000000000000000 P f0e46a3c1e7546d8b03a472ef7c0d5b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:53:22.478840 26673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0e46a3c1e7546d8b03a472ef7c0d5b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:53:22.479015 26668 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0e46a3c1e7546d8b03a472ef7c0d5b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:53:22.479035 26673 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0e46a3c1e7546d8b03a472ef7c0d5b5 [term 1 LEADER]: Becoming Leader. State: Replica: f0e46a3c1e7546d8b03a472ef7c0d5b5, State: Running, Role: LEADER
I20260928 08:53:22.479197 26673 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0e46a3c1e7546d8b03a472ef7c0d5b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e46a3c1e7546d8b03a472ef7c0d5b5" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38151 } }
I20260928 08:53:22.479600 26675 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0e46a3c1e7546d8b03a472ef7c0d5b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0e46a3c1e7546d8b03a472ef7c0d5b5. Latest consensus state: current_term: 1 leader_uuid: "f0e46a3c1e7546d8b03a472ef7c0d5b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e46a3c1e7546d8b03a472ef7c0d5b5" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38151 } } }
I20260928 08:53:22.479674 26675 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0e46a3c1e7546d8b03a472ef7c0d5b5 [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:22.479815 26674 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0e46a3c1e7546d8b03a472ef7c0d5b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0e46a3c1e7546d8b03a472ef7c0d5b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e46a3c1e7546d8b03a472ef7c0d5b5" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38151 } } }
I20260928 08:53:22.479861 26674 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0e46a3c1e7546d8b03a472ef7c0d5b5 [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:22.480255 26682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:53:22.480760 26682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:53:22.482364 26682 catalog_manager.cc:1347] Generated new cluster ID: d312905869fd4f509a08aa4a3d925bd1
I20260928 08:53:22.482414 26682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 08:53:22 dist-test-slave-pbjp krb5kdc[26587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:22.486598 26672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:22.475956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33270 (local address 127.23.167.254:38151)
0928 08:53:22.476151 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:22.476160 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:22.476192 (+ 32us) server_negotiation.cc:407] Connection header received
0928 08:53:22.476597 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:22.476606 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:22.476667 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:22.476828 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:22.477415 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:22.479046 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:22.480206 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:22.480777 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:22.480875 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:22.483295 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:22.483300 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:22.483300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:22.483311 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:22.485100 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:22.485466 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:22.485468 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:22.485469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:22.485562 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:22.485748 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:22.485749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:22.485749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:22.486017 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:22.486151 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:22.486288 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 08:53:22.486409 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":81,"threads_started":1}
I20260928 08:53:22.496011 26682 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:53:22.496747 26682 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:53:22.499485 26682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0e46a3c1e7546d8b03a472ef7c0d5b5: Generated new TSK 0
I20260928 08:53:22.499648 26682 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.167.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
I20260928 08:53:22.527127 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:0
--local_ip_for_outbound_sockets=127.23.167.193
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:38151
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:22.607285 26694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:22.607527 26694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:22.607563 26694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:22.608834 26694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:22.608882 26694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:22.608922 26694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:53:22.610469 26694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:38151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:22.610723 26694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:22.610944 26694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:22.615821 26700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:22.615826 26702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:22.616004 26694 server_base.cc:1034] running on GCE node
W20260928 08:53:22.615875 26699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:22.616279 26694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:22.616540 26694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:22.617673 26694 hybrid_clock.cc:648] HybridClock initialized: now 1790585602617658 us; error 30 us; skew 500 ppm
Sep 28 08:53:22 dist-test-slave-pbjp krb5kdc[26587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585602, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:22.620138 26694 init.cc:377] Logged in from keytab as kudu/127.23.167.193@KRBTEST.COM (short username kudu)
I20260928 08:53:22.620600 26694 webserver.cc:460] Webserver started at http://127.23.167.193:42795/ using document root <none> and password file <none>
I20260928 08:53:22.620817 26694 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:22.620867 26694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:22.620986 26694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:22.622071 26694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/data/instance:
uuid: "da70309a1235495aab9f5d769109d48f"
format_stamp: "Formatted at 2026-09-28 08:53:22 on dist-test-slave-pbjp"
server_key: "1fc579ea8d86bc4ff584fac67a7c50c7"
server_key_iv: "b3cd2235140b617e280f6d8b06a70050"
server_key_version: "encryptionkey@0"
I20260928 08:53:22.622402 26694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance:
uuid: "da70309a1235495aab9f5d769109d48f"
format_stamp: "Formatted at 2026-09-28 08:53:22 on dist-test-slave-pbjp"
server_key: "1fc579ea8d86bc4ff584fac67a7c50c7"
server_key_iv: "b3cd2235140b617e280f6d8b06a70050"
server_key_version: "encryptionkey@0"
I20260928 08:53:22.623629 26694 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:53:22.624328 26709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:22.624502 26694 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:53:22.624585 26694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "da70309a1235495aab9f5d769109d48f"
format_stamp: "Formatted at 2026-09-28 08:53:22 on dist-test-slave-pbjp"
server_key: "1fc579ea8d86bc4ff584fac67a7c50c7"
server_key_iv: "b3cd2235140b617e280f6d8b06a70050"
server_key_version: "encryptionkey@0"
I20260928 08:53:22.624646 26694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:22.635819 26694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:22.637609 26694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:22.637738 26694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:22.637933 26694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:53:22.638209 26694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:22.638242 26694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:22.638271 26694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:22.638295 26694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:22.643862 26694 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:39081
I20260928 08:53:22.643918 26822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:39081 every 8 connection(s)
I20260928 08:53:22.644224 26694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
Sep 28 08:53:22 dist-test-slave-pbjp krb5kdc[26587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585602, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:22.652179 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 26694
I20260928 08:53:22.652264 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance
I20260928 08:53:22.652420 24223 external_mini_cluster.cc:1428] Setting key 35ef53c0a7ac9665dfaed0ec50567aed
I20260928 08:53:22.652951 26672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:22.644932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:54361 (local address 127.23.167.254:38151)
0928 08:53:22.644984 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:22.644986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:22.645148 (+ 162us) server_negotiation.cc:407] Connection header received
0928 08:53:22.645655 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:22.645656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:22.645667 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:22.645726 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:22.646688 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:22.647202 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:22.648151 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:22.648360 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:22.648435 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:22.650458 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:22.650459 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:22.650459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:22.650467 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:22.651671 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:22.652057 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:22.652058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:22.652059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:22.652109 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:22.652356 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:22.652357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:22.652358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:22.652481 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:22.652542 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:22.652813 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 08:53:22.652883 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 08:53:22.653223 26825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:22.644861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38151 (local address 127.23.167.193:54361)
0928 08:53:22.645010 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:22.645054 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:22.645561 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:22.645790 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:22.645794 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:22.646395 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:22.646597 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:22.646602 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:22.647270 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:22.647271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:22.648062 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:22.648065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:22.648446 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:22.648446 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:22.648541 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:22.649013 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:22.649022 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:22.650363 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:22.651745 (+ 1382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:22.651752 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:22.651754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:22.651973 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:22.652165 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:22.652165 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:22.652166 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:22.652274 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:22.652541 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:22.652548 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:22.652681 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 08:53:22.652799 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 08:53:22.652964 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260928 08:53:22.653729 26823 heartbeater.cc:347] Connected to a master server at 127.23.167.254:38151
I20260928 08:53:22.653815 26823 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:22.653995 26823 heartbeater.cc:511] Master 127.23.167.254:38151 requested a full tablet report, sending...
I20260928 08:53:22.654520 26632 ts_manager.cc:194] Registered new tserver with Master: da70309a1235495aab9f5d769109d48f (127.23.167.193:39081)
I20260928 08:53:22.655134 26632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.193@KRBTEST.COM'} at 127.23.167.193:54361
WARNING: no policy specified for kudu/127.23.167.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
I20260928 08:53:22.674119 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:0
--local_ip_for_outbound_sockets=127.23.167.194
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:38151
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:22.754256 26828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:22.754470 26828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:22.754501 26828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:22.755661 26828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:22.755705 26828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:22.755740 26828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:53:22.757225 26828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:38151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:22.757471 26828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:22.757685 26828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:22.762598 26834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:22.762642 26828 server_base.cc:1034] running on GCE node
W20260928 08:53:22.762607 26833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:22.762605 26836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:22.762954 26828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:22.763185 26828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:22.764341 26828 hybrid_clock.cc:648] HybridClock initialized: now 1790585602764323 us; error 29 us; skew 500 ppm
Sep 28 08:53:22 dist-test-slave-pbjp krb5kdc[26587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585602, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:22.766773 26828 init.cc:377] Logged in from keytab as kudu/127.23.167.194@KRBTEST.COM (short username kudu)
I20260928 08:53:22.767184 26828 webserver.cc:460] Webserver started at http://127.23.167.194:41753/ using document root <none> and password file <none>
I20260928 08:53:22.767406 26828 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:22.767457 26828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:22.767580 26828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:22.768667 26828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/data/instance:
uuid: "a02ed4f794dd4f2aadc18face373ba14"
format_stamp: "Formatted at 2026-09-28 08:53:22 on dist-test-slave-pbjp"
server_key: "a9630f0e7116ed4012229a204daa89c5"
server_key_iv: "132835666619dbfb2a1597834d1f5cb6"
server_key_version: "encryptionkey@0"
I20260928 08:53:22.768999 26828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance:
uuid: "a02ed4f794dd4f2aadc18face373ba14"
format_stamp: "Formatted at 2026-09-28 08:53:22 on dist-test-slave-pbjp"
server_key: "a9630f0e7116ed4012229a204daa89c5"
server_key_iv: "132835666619dbfb2a1597834d1f5cb6"
server_key_version: "encryptionkey@0"
I20260928 08:53:22.770238 26828 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:53:22.770939 26843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:22.771073 26828 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:53:22.771154 26828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "a02ed4f794dd4f2aadc18face373ba14"
format_stamp: "Formatted at 2026-09-28 08:53:22 on dist-test-slave-pbjp"
server_key: "a9630f0e7116ed4012229a204daa89c5"
server_key_iv: "132835666619dbfb2a1597834d1f5cb6"
server_key_version: "encryptionkey@0"
I20260928 08:53:22.771212 26828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:22.806839 26828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:22.809098 26828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:22.809257 26828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:22.809491 26828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:53:22.809850 26828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:22.809885 26828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:22.809916 26828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:22.809933 26828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:22.815326 26828 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:45937
I20260928 08:53:22.815382 26956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:45937 every 8 connection(s)
I20260928 08:53:22.815696 26828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
I20260928 08:53:22.818645 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 26828
I20260928 08:53:22.818727 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance
I20260928 08:53:22.818892 24223 external_mini_cluster.cc:1428] Setting key 834925245b3cc76a3808b00a6780a3ef
Sep 28 08:53:22 dist-test-slave-pbjp krb5kdc[26587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585602, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:22.825618 26672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:22.816506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:35533 (local address 127.23.167.254:38151)
0928 08:53:22.816575 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:22.816577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:22.816697 (+ 120us) server_negotiation.cc:407] Connection header received
0928 08:53:22.817277 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:22.817278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:22.817291 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:22.817337 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:22.818298 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:22.818845 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:22.820097 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:22.820299 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:22.820356 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:22.822758 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:22.822760 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:22.822760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:22.822768 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:22.824139 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:22.824584 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:22.824586 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:22.824586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:22.824644 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:22.824869 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:22.824870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:22.824870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:22.825001 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:22.825057 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:22.825425 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 08:53:22.825514 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 08:53:22.825718 26959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:22.816398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38151 (local address 127.23.167.194:35533)
0928 08:53:22.816563 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:22.816615 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:22.817169 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:22.817443 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:22.817447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:22.818003 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:22.818200 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:22.818205 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:22.818948 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:22.818949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:22.820015 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:22.820019 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:22.820376 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:22.820377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:22.820509 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:22.821031 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:22.821040 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:22.822654 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:22.824232 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:22.824240 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:22.824242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:22.824509 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:22.824701 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:22.824701 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:22.824702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:22.824812 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:22.825056 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:22.825086 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:22.825233 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 08:53:22.825367 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 08:53:22.825529 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260928 08:53:22.826236 26957 heartbeater.cc:347] Connected to a master server at 127.23.167.254:38151
I20260928 08:53:22.826331 26957 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:22.826516 26957 heartbeater.cc:511] Master 127.23.167.254:38151 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.167.195@KRBTEST.COM; defaulting to no policy
I20260928 08:53:22.826911 26632 ts_manager.cc:194] Registered new tserver with Master: a02ed4f794dd4f2aadc18face373ba14 (127.23.167.194:45937)
I20260928 08:53:22.827272 26632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.194@KRBTEST.COM'} at 127.23.167.194:35533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
I20260928 08:53:22.841840 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:0
--local_ip_for_outbound_sockets=127.23.167.195
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:38151
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:22.921008 26962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:22.921271 26962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:22.921305 26962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:22.922539 26962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:22.922586 26962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:22.922623 26962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:53:22.924104 26962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:38151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:22.924345 26962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:22.924561 26962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:22.929507 26968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:22.929513 26970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:22.929737 26962 server_base.cc:1034] running on GCE node
W20260928 08:53:22.929522 26967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:22.930011 26962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:22.930222 26962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:22.931358 26962 hybrid_clock.cc:648] HybridClock initialized: now 1790585602931341 us; error 33 us; skew 500 ppm
Sep 28 08:53:22 dist-test-slave-pbjp krb5kdc[26587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585602, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:22.933876 26962 init.cc:377] Logged in from keytab as kudu/127.23.167.195@KRBTEST.COM (short username kudu)
I20260928 08:53:22.934377 26962 webserver.cc:460] Webserver started at http://127.23.167.195:34297/ using document root <none> and password file <none>
I20260928 08:53:22.934600 26962 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:22.934648 26962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:22.934769 26962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:22.935794 26962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/data/instance:
uuid: "e19080e1fa554a52966751ebfbe6d523"
format_stamp: "Formatted at 2026-09-28 08:53:22 on dist-test-slave-pbjp"
server_key: "50d35aa8a5666ef1b271003aa1b55ee0"
server_key_iv: "01b2665465161671d4ad3bb886128768"
server_key_version: "encryptionkey@0"
I20260928 08:53:22.936125 26962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance:
uuid: "e19080e1fa554a52966751ebfbe6d523"
format_stamp: "Formatted at 2026-09-28 08:53:22 on dist-test-slave-pbjp"
server_key: "50d35aa8a5666ef1b271003aa1b55ee0"
server_key_iv: "01b2665465161671d4ad3bb886128768"
server_key_version: "encryptionkey@0"
I20260928 08:53:22.937391 26962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 08:53:22.938092 26977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:22.938242 26962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:53:22.938323 26962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "e19080e1fa554a52966751ebfbe6d523"
format_stamp: "Formatted at 2026-09-28 08:53:22 on dist-test-slave-pbjp"
server_key: "50d35aa8a5666ef1b271003aa1b55ee0"
server_key_iv: "01b2665465161671d4ad3bb886128768"
server_key_version: "encryptionkey@0"
I20260928 08:53:22.938382 26962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:22.946506 26962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:22.948338 26962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:22.948467 26962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:22.948658 26962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:53:22.948942 26962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:22.948977 26962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:22.949005 26962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:22.949028 26962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:22.954487 26962 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:37795
I20260928 08:53:22.954540 27090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:37795 every 8 connection(s)
I20260928 08:53:22.954835 26962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
I20260928 08:53:22.956151 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 26962
I20260928 08:53:22.956245 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance
I20260928 08:53:22.956439 24223 external_mini_cluster.cc:1428] Setting key 7af970828f4c44db985b2a108b9f74ca
Sep 28 08:53:22 dist-test-slave-pbjp krb5kdc[26587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585602, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:22.963838 26672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:22.955597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:48005 (local address 127.23.167.254:38151)
0928 08:53:22.955674 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:22.955676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:22.955826 (+ 150us) server_negotiation.cc:407] Connection header received
0928 08:53:22.956393 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:22.956394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:22.956409 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:22.956479 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:22.957640 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:22.958135 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:22.959131 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:22.959385 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:22.959442 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:22.961494 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:22.961496 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:22.961496 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:22.961503 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:22.962739 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:22.963055 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:22.963056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:22.963056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:22.963102 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:22.963302 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:22.963303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:22.963303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:22.963402 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:22.963449 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:22.963704 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 08:53:22.963771 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 08:53:22.963935 27093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:22.955546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38151 (local address 127.23.167.195:48005)
0928 08:53:22.955707 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:22.955744 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:22.956222 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:22.956559 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:22.956564 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:22.957346 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:22.957562 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:22.957567 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:22.958220 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:22.958221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:22.959035 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:22.959038 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:22.959464 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:22.959465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:22.959572 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:22.960042 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:22.960050 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:22.961376 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:22.962807 (+ 1431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:22.962811 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:22.962813 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:22.963006 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:22.963145 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:22.963145 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:22.963146 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:22.963231 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:22.963448 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:22.963451 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:22.963564 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 08:53:22.963639 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 08:53:22.963756 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260928 08:53:22.964370 27091 heartbeater.cc:347] Connected to a master server at 127.23.167.254:38151
I20260928 08:53:22.964484 27091 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:22.964659 27091 heartbeater.cc:511] Master 127.23.167.254:38151 requested a full tablet report, sending...
I20260928 08:53:22.965107 26632 ts_manager.cc:194] Registered new tserver with Master: e19080e1fa554a52966751ebfbe6d523 (127.23.167.195:37795)
I20260928 08:53:22.965487 26632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.195@KRBTEST.COM'} at 127.23.167.195:48005
I20260928 08:53:22.967999 24223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 08:53:22.978451 26672 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:53:22.977556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44528 (local address 127.23.167.254:38151)
0928 08:53:22.977605 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:22.977607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:22.977695 (+ 88us) server_negotiation.cc:407] Connection header received
0928 08:53:22.977771 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:22.977772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:22.977791 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:22.977837 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:22.978356 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44528: BlockingRecv error: recv got EOF from 127.0.0.1:44528 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 08:53:22.979274 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 26694
I20260928 08:53:22.983352 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 26828
I20260928 08:53:22.987375 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 26962
I20260928 08:53:22.991161 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 26600
2026-09-28T08:53:22Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1720 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:53:23 dist-test-slave-pbjp krb5kdc[27103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:53:23 dist-test-slave-pbjp krb5kdc[27103](info): set up 2 sockets
Sep 28 08:53:23 dist-test-slave-pbjp krb5kdc[27103](info): commencing operation
krb5kdc: starting...
W20260928 08:53:23.936637 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:53:24 dist-test-slave-pbjp krb5kdc[27103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:53:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:53:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.167.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
I20260928 08:53:24.051993 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:46627
--webserver_interface=127.23.167.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:46627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:24.131477 27116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:24.131713 27116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:24.131745 27116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:24.132946 27116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:53:24.132994 27116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:24.133013 27116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:24.133028 27116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:53:24.133044 27116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:53:24.134678 27116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:46627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:46627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:24.134961 27116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:24.135188 27116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:24.140177 27121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:24.140177 27122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:24.140209 27124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:24.140202 27116 server_base.cc:1034] running on GCE node
I20260928 08:53:24.140642 27116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:24.140931 27116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:24.142098 27116 hybrid_clock.cc:648] HybridClock initialized: now 1790585604142088 us; error 45 us; skew 500 ppm
Sep 28 08:53:24 dist-test-slave-pbjp krb5kdc[27103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585604, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:24.144660 27116 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:53:24.145164 27116 webserver.cc:460] Webserver started at http://127.23.167.254:45315/ using document root <none> and password file <none>
I20260928 08:53:24.145390 27116 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:24.145438 27116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:24.145558 27116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:24.146685 27116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "eef8aa4c515e472ea062b492a2ccf68b"
format_stamp: "Formatted at 2026-09-28 08:53:24 on dist-test-slave-pbjp"
server_key: "93ca0946cf376f14887283350796f149"
server_key_iv: "f749773e81fc22bcde59eb9b6c92152e"
server_key_version: "encryptionkey@0"
I20260928 08:53:24.147033 27116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "eef8aa4c515e472ea062b492a2ccf68b"
format_stamp: "Formatted at 2026-09-28 08:53:24 on dist-test-slave-pbjp"
server_key: "93ca0946cf376f14887283350796f149"
server_key_iv: "f749773e81fc22bcde59eb9b6c92152e"
server_key_version: "encryptionkey@0"
I20260928 08:53:24.148419 27116 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:53:24.149202 27131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:24.149403 27116 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:53:24.149492 27116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "eef8aa4c515e472ea062b492a2ccf68b"
format_stamp: "Formatted at 2026-09-28 08:53:24 on dist-test-slave-pbjp"
server_key: "93ca0946cf376f14887283350796f149"
server_key_iv: "f749773e81fc22bcde59eb9b6c92152e"
server_key_version: "encryptionkey@0"
I20260928 08:53:24.149560 27116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:24.163805 27116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:24.166081 27116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:24.166224 27116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:24.170094 27116 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:46627
I20260928 08:53:24.170141 27183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:46627 every 8 connection(s)
I20260928 08:53:24.170621 27116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:53:24.171329 27184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:24.173689 27184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eef8aa4c515e472ea062b492a2ccf68b: Bootstrap starting.
I20260928 08:53:24.174248 27184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eef8aa4c515e472ea062b492a2ccf68b: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:24.174479 27184 log.cc:824] T 00000000000000000000000000000000 P eef8aa4c515e472ea062b492a2ccf68b: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:24.175086 27184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eef8aa4c515e472ea062b492a2ccf68b: No bootstrap required, opened a new log
I20260928 08:53:24.176746 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 27116
I20260928 08:53:24.176620 27184 raft_consensus.cc:348] T 00000000000000000000000000000000 P eef8aa4c515e472ea062b492a2ccf68b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eef8aa4c515e472ea062b492a2ccf68b" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46627 } }
I20260928 08:53:24.176805 27184 raft_consensus.cc:374] T 00000000000000000000000000000000 P eef8aa4c515e472ea062b492a2ccf68b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:24.176844 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:53:24.176839 27184 raft_consensus.cc:729] T 00000000000000000000000000000000 P eef8aa4c515e472ea062b492a2ccf68b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eef8aa4c515e472ea062b492a2ccf68b, State: Initialized, Role: FOLLOWER
I20260928 08:53:24.177009 24223 external_mini_cluster.cc:1428] Setting key b9e0236ce51d453ea258a91f2dbcdb63
I20260928 08:53:24.176987 27184 consensus_queue.cc:260] T 00000000000000000000000000000000 P eef8aa4c515e472ea062b492a2ccf68b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eef8aa4c515e472ea062b492a2ccf68b" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46627 } }
I20260928 08:53:24.177079 27184 raft_consensus.cc:388] T 00000000000000000000000000000000 P eef8aa4c515e472ea062b492a2ccf68b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:53:24.177121 27184 raft_consensus.cc:482] T 00000000000000000000000000000000 P eef8aa4c515e472ea062b492a2ccf68b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:53:24.177169 27184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eef8aa4c515e472ea062b492a2ccf68b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:24.178463 27184 raft_consensus.cc:504] T 00000000000000000000000000000000 P eef8aa4c515e472ea062b492a2ccf68b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eef8aa4c515e472ea062b492a2ccf68b" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46627 } }
I20260928 08:53:24.178573 27184 leader_election.cc:302] T 00000000000000000000000000000000 P eef8aa4c515e472ea062b492a2ccf68b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eef8aa4c515e472ea062b492a2ccf68b; no voters:
I20260928 08:53:24.178738 27184 leader_election.cc:288] T 00000000000000000000000000000000 P eef8aa4c515e472ea062b492a2ccf68b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:53:24.178771 27189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eef8aa4c515e472ea062b492a2ccf68b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:53:24.178941 27189 raft_consensus.cc:686] T 00000000000000000000000000000000 P eef8aa4c515e472ea062b492a2ccf68b [term 1 LEADER]: Becoming Leader. State: Replica: eef8aa4c515e472ea062b492a2ccf68b, State: Running, Role: LEADER
I20260928 08:53:24.178998 27184 sys_catalog.cc:564] T 00000000000000000000000000000000 P eef8aa4c515e472ea062b492a2ccf68b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:53:24.179031 27189 consensus_queue.cc:237] T 00000000000000000000000000000000 P eef8aa4c515e472ea062b492a2ccf68b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eef8aa4c515e472ea062b492a2ccf68b" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46627 } }
I20260928 08:53:24.179543 27190 sys_catalog.cc:455] T 00000000000000000000000000000000 P eef8aa4c515e472ea062b492a2ccf68b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eef8aa4c515e472ea062b492a2ccf68b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eef8aa4c515e472ea062b492a2ccf68b" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46627 } } }
I20260928 08:53:24.179556 27191 sys_catalog.cc:455] T 00000000000000000000000000000000 P eef8aa4c515e472ea062b492a2ccf68b [sys.catalog]: SysCatalogTable state changed. Reason: New leader eef8aa4c515e472ea062b492a2ccf68b. Latest consensus state: current_term: 1 leader_uuid: "eef8aa4c515e472ea062b492a2ccf68b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eef8aa4c515e472ea062b492a2ccf68b" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46627 } } }
I20260928 08:53:24.179705 27190 sys_catalog.cc:458] T 00000000000000000000000000000000 P eef8aa4c515e472ea062b492a2ccf68b [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:24.179768 27191 sys_catalog.cc:458] T 00000000000000000000000000000000 P eef8aa4c515e472ea062b492a2ccf68b [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:24.180342 27198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:53:24.180743 27198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:53:24.183260 27198 catalog_manager.cc:1347] Generated new cluster ID: 71558de566c448458eb25d796f86767d
I20260928 08:53:24.183310 27198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 08:53:24 dist-test-slave-pbjp krb5kdc[27103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:24.187750 27188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:24.177505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40114 (local address 127.23.167.254:46627)
0928 08:53:24.177667 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:24.177677 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:24.177703 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:53:24.177824 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:24.177830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:24.177872 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:24.177983 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:24.178733 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:24.180188 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:24.181161 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:24.181866 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:24.182105 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:24.184367 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:24.184371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:24.184371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:24.184383 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:24.186226 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:24.186602 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:24.186605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:24.186605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:24.186719 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:24.186912 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:24.186912 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:24.186913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:24.187177 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:24.187329 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:24.187420 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 08:53:24.187544 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260928 08:53:24.188853 24223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:53:24.199873 27198 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:53:24.200593 27198 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:53:24.204725 27188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:24.200447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40118 (local address 127.23.167.254:46627)
0928 08:53:24.200501 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:24.200503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:24.200661 (+ 158us) server_negotiation.cc:407] Connection header received
0928 08:53:24.200742 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:24.200744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:24.200754 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:24.200792 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:24.201308 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:24.202033 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:24.203140 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:24.203452 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:24.203543 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:24.204235 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:24.204236 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:24.204237 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:53:24.204479 (+ 242us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:24.204498 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:53:24.204505 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:24.204554 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:24.204620 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 08:53:24.204646 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 08:53:24.206979 27198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eef8aa4c515e472ea062b492a2ccf68b: Generated new TSK 0
I20260928 08:53:24.207145 27198 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:53:24.213409 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 27116
2026-09-28T08:53:24Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1220 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:53:24 dist-test-slave-pbjp krb5kdc[27217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:53:24 dist-test-slave-pbjp krb5kdc[27217](info): set up 2 sockets
Sep 28 08:53:24 dist-test-slave-pbjp krb5kdc[27217](info): commencing operation
krb5kdc: starting...
W20260928 08:53:25.168722 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:53:25 dist-test-slave-pbjp krb5kdc[27217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:53:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:53:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.167.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
I20260928 08:53:25.280365 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:34515
--webserver_interface=127.23.167.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:44569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:34515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:25.361923 27230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:25.362145 27230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:25.362176 27230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:25.363406 27230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:53:25.363456 27230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:25.363474 27230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:25.363490 27230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:53:25.363504 27230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:53:25.365024 27230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:44569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:34515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:34515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:25.365269 27230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:25.365453 27230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:25.370200 27236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:25.370180 27235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:25.370177 27238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:25.370265 27230 server_base.cc:1034] running on GCE node
I20260928 08:53:25.370611 27230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:25.370893 27230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:25.372056 27230 hybrid_clock.cc:648] HybridClock initialized: now 1790585605372037 us; error 33 us; skew 500 ppm
Sep 28 08:53:25 dist-test-slave-pbjp krb5kdc[27217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585605, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:25.374614 27230 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:53:25.375066 27230 webserver.cc:460] Webserver started at http://127.23.167.254:43033/ using document root <none> and password file <none>
I20260928 08:53:25.375283 27230 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:25.375332 27230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:25.375463 27230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:25.376523 27230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "0b4e07dece8f4413b09e4a8d321f241a"
format_stamp: "Formatted at 2026-09-28 08:53:25 on dist-test-slave-pbjp"
server_key: "37424c1756be637477ff61c1998298e7"
server_key_iv: "14f42bc172d67a82eff95d1248c9b096"
server_key_version: "encryptionkey@0"
I20260928 08:53:25.376844 27230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "0b4e07dece8f4413b09e4a8d321f241a"
format_stamp: "Formatted at 2026-09-28 08:53:25 on dist-test-slave-pbjp"
server_key: "37424c1756be637477ff61c1998298e7"
server_key_iv: "14f42bc172d67a82eff95d1248c9b096"
server_key_version: "encryptionkey@0"
I20260928 08:53:25.378149 27230 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 08:53:25.378901 27245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:25.379048 27230 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:53:25.379133 27230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "0b4e07dece8f4413b09e4a8d321f241a"
format_stamp: "Formatted at 2026-09-28 08:53:25 on dist-test-slave-pbjp"
server_key: "37424c1756be637477ff61c1998298e7"
server_key_iv: "14f42bc172d67a82eff95d1248c9b096"
server_key_version: "encryptionkey@0"
I20260928 08:53:25.379204 27230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:25.388659 27230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:25.390549 27230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:25.390669 27230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:25.394287 27230 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:34515
I20260928 08:53:25.394338 27297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:34515 every 8 connection(s)
I20260928 08:53:25.394629 27230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:53:25.394760 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 27230
I20260928 08:53:25.394866 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:53:25.395044 24223 external_mini_cluster.cc:1428] Setting key 1d68663d7c94495e5dd54bebb3a8b2cd
I20260928 08:53:25.395669 27298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:25.398096 27298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b4e07dece8f4413b09e4a8d321f241a: Bootstrap starting.
I20260928 08:53:25.398628 27298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b4e07dece8f4413b09e4a8d321f241a: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:25.398880 27298 log.cc:824] T 00000000000000000000000000000000 P 0b4e07dece8f4413b09e4a8d321f241a: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:25.399469 27298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b4e07dece8f4413b09e4a8d321f241a: No bootstrap required, opened a new log
Sep 28 08:53:25 dist-test-slave-pbjp krb5kdc[27217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:25.401273 27298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b4e07dece8f4413b09e4a8d321f241a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4e07dece8f4413b09e4a8d321f241a" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34515 } }
I20260928 08:53:25.401407 27298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b4e07dece8f4413b09e4a8d321f241a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:25.401438 27298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b4e07dece8f4413b09e4a8d321f241a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b4e07dece8f4413b09e4a8d321f241a, State: Initialized, Role: FOLLOWER
I20260928 08:53:25.401553 27298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b4e07dece8f4413b09e4a8d321f241a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4e07dece8f4413b09e4a8d321f241a" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34515 } }
I20260928 08:53:25.401619 27298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b4e07dece8f4413b09e4a8d321f241a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:53:25.401648 27298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b4e07dece8f4413b09e4a8d321f241a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:53:25.401700 27298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b4e07dece8f4413b09e4a8d321f241a [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:25.402181 27298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b4e07dece8f4413b09e4a8d321f241a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4e07dece8f4413b09e4a8d321f241a" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34515 } }
I20260928 08:53:25.402266 27298 leader_election.cc:302] T 00000000000000000000000000000000 P 0b4e07dece8f4413b09e4a8d321f241a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b4e07dece8f4413b09e4a8d321f241a; no voters:
I20260928 08:53:25.402447 27298 leader_election.cc:288] T 00000000000000000000000000000000 P 0b4e07dece8f4413b09e4a8d321f241a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:53:25.402478 27303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b4e07dece8f4413b09e4a8d321f241a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:53:25.402645 27303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b4e07dece8f4413b09e4a8d321f241a [term 1 LEADER]: Becoming Leader. State: Replica: 0b4e07dece8f4413b09e4a8d321f241a, State: Running, Role: LEADER
I20260928 08:53:25.402693 27298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b4e07dece8f4413b09e4a8d321f241a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:53:25.403952 27303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b4e07dece8f4413b09e4a8d321f241a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4e07dece8f4413b09e4a8d321f241a" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34515 } }
I20260928 08:53:25.404448 27304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b4e07dece8f4413b09e4a8d321f241a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b4e07dece8f4413b09e4a8d321f241a. Latest consensus state: current_term: 1 leader_uuid: "0b4e07dece8f4413b09e4a8d321f241a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4e07dece8f4413b09e4a8d321f241a" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34515 } } }
I20260928 08:53:25.404546 27304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b4e07dece8f4413b09e4a8d321f241a [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:25.404675 27301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:25.395584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35360 (local address 127.23.167.254:34515)
0928 08:53:25.395740 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:25.395750 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:25.395780 (+ 30us) server_negotiation.cc:407] Connection header received
0928 08:53:25.396210 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:25.396217 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:25.396262 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:25.396391 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:25.396977 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:25.397919 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:25.398671 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:25.399131 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:25.399247 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:25.401394 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:25.401398 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:25.401399 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:25.401412 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:25.403146 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:25.403525 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:25.403527 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:25.403528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:25.403609 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:25.403811 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:25.403812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:25.403813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:25.404056 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:25.404183 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:25.404313 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 08:53:25.404446 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260928 08:53:25.404767 27305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b4e07dece8f4413b09e4a8d321f241a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b4e07dece8f4413b09e4a8d321f241a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4e07dece8f4413b09e4a8d321f241a" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34515 } } }
I20260928 08:53:25.404835 27305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b4e07dece8f4413b09e4a8d321f241a [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:25.405236 27319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:53:25.405678 27319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:53:25.406929 27319 catalog_manager.cc:1347] Generated new cluster ID: 4c07b06cef414a0caa616cef25cd9331
I20260928 08:53:25.406978 27319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:53:25.413609 27319 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:53:25.414132 27319 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:53:25.420995 27319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b4e07dece8f4413b09e4a8d321f241a: Generated new TSK 0
I20260928 08:53:25.421202 27319 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.167.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
I20260928 08:53:25.449882 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:0
--local_ip_for_outbound_sockets=127.23.167.193
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:34515
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:44569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:25.530300 27325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:25.530553 27325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:25.530589 27325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:25.531776 27325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:25.531836 27325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:25.531863 27325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:53:25.533404 27325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:44569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:34515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:25.533627 27325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:25.533872 27325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:25.538991 27331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:25.538991 27333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:25.539049 27330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:25.539019 27325 server_base.cc:1034] running on GCE node
I20260928 08:53:25.539391 27325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:25.539603 27325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:25.540761 27325 hybrid_clock.cc:648] HybridClock initialized: now 1790585605540741 us; error 34 us; skew 500 ppm
Sep 28 08:53:25 dist-test-slave-pbjp krb5kdc[27217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585605, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:25.543381 27325 init.cc:377] Logged in from keytab as kudu/127.23.167.193@KRBTEST.COM (short username kudu)
I20260928 08:53:25.543872 27325 webserver.cc:460] Webserver started at http://127.23.167.193:43935/ using document root <none> and password file <none>
I20260928 08:53:25.544077 27325 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:25.544126 27325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:25.544250 27325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:25.545375 27325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/data/instance:
uuid: "37b3e0853dc44be2a116b61ab31b1d13"
format_stamp: "Formatted at 2026-09-28 08:53:25 on dist-test-slave-pbjp"
server_key: "9c2dac536518853408b31cccdccb374e"
server_key_iv: "3136f9657815f9b7f32c839e1dc809a9"
server_key_version: "encryptionkey@0"
I20260928 08:53:25.545712 27325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance:
uuid: "37b3e0853dc44be2a116b61ab31b1d13"
format_stamp: "Formatted at 2026-09-28 08:53:25 on dist-test-slave-pbjp"
server_key: "9c2dac536518853408b31cccdccb374e"
server_key_iv: "3136f9657815f9b7f32c839e1dc809a9"
server_key_version: "encryptionkey@0"
I20260928 08:53:25.546965 27325 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:53:25.547679 27340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:25.547838 27325 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:53:25.547921 27325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "37b3e0853dc44be2a116b61ab31b1d13"
format_stamp: "Formatted at 2026-09-28 08:53:25 on dist-test-slave-pbjp"
server_key: "9c2dac536518853408b31cccdccb374e"
server_key_iv: "3136f9657815f9b7f32c839e1dc809a9"
server_key_version: "encryptionkey@0"
I20260928 08:53:25.547982 27325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:25.562285 27325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:25.564109 27325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:25.564248 27325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:25.564476 27325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:53:25.564785 27325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:25.564817 27325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:25.564847 27325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:25.564869 27325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:25.570587 27325 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:42493
I20260928 08:53:25.570634 27453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:42493 every 8 connection(s)
I20260928 08:53:25.570945 27325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
I20260928 08:53:25.574637 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 27325
I20260928 08:53:25.574728 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance
I20260928 08:53:25.574884 24223 external_mini_cluster.cc:1428] Setting key b60786794f32af1e229936e6f6e11d64
Sep 28 08:53:25 dist-test-slave-pbjp krb5kdc[27217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585605, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:25.580132 27301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:25.571617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:34045 (local address 127.23.167.254:34515)
0928 08:53:25.571688 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:25.571690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:25.571866 (+ 176us) server_negotiation.cc:407] Connection header received
0928 08:53:25.572344 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:25.572345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:25.572355 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:25.572405 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:25.573345 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:25.573849 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:25.574991 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:25.575272 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:25.575334 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:25.577509 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:25.577511 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:25.577511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:25.577519 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:25.578752 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:25.579221 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:25.579222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:25.579222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:25.579281 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:25.579534 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:25.579534 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:25.579535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:25.579628 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:25.579673 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:25.579999 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 08:53:25.580065 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 08:53:25.580404 27456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:25.571587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34515 (local address 127.23.167.193:34045)
0928 08:53:25.571740 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:25.571776 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:25.572267 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:25.572469 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:25.572472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:25.573017 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:25.573261 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:25.573266 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:25.573947 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:25.573949 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:25.574900 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:25.574903 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:25.575365 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:25.575366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:25.575510 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:25.575958 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:25.575966 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:25.577419 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:25.578842 (+ 1423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:25.578850 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:25.578853 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:25.579148 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:25.579335 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:25.579336 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:25.579336 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:25.579476 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:25.579682 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:25.579688 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:25.579870 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 08:53:25.579991 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 08:53:25.580160 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20260928 08:53:25.580919 27454 heartbeater.cc:347] Connected to a master server at 127.23.167.254:34515
I20260928 08:53:25.581013 27454 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:25.581212 27454 heartbeater.cc:511] Master 127.23.167.254:34515 requested a full tablet report, sending...
I20260928 08:53:25.581835 27262 ts_manager.cc:194] Registered new tserver with Master: 37b3e0853dc44be2a116b61ab31b1d13 (127.23.167.193:42493)
WARNING: no policy specified for kudu/127.23.167.194@KRBTEST.COM; defaulting to no policy
I20260928 08:53:25.582784 27262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.193@KRBTEST.COM'} at 127.23.167.193:34045
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
I20260928 08:53:25.599766 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:0
--local_ip_for_outbound_sockets=127.23.167.194
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:34515
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:44569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:25.680807 27459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:25.681012 27459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:25.681034 27459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:25.682170 27459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:25.682210 27459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:25.682235 27459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:53:25.683612 27459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:44569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:34515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:25.683800 27459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:25.684001 27459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:25.688827 27467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:25.688856 27459 server_base.cc:1034] running on GCE node
W20260928 08:53:25.688827 27464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:25.688954 27465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:25.689277 27459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:25.689483 27459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:25.690623 27459 hybrid_clock.cc:648] HybridClock initialized: now 1790585605690608 us; error 29 us; skew 500 ppm
Sep 28 08:53:25 dist-test-slave-pbjp krb5kdc[27217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585605, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:25.692899 27459 init.cc:377] Logged in from keytab as kudu/127.23.167.194@KRBTEST.COM (short username kudu)
I20260928 08:53:25.693385 27459 webserver.cc:460] Webserver started at http://127.23.167.194:38563/ using document root <none> and password file <none>
I20260928 08:53:25.693583 27459 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:25.693624 27459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:25.693719 27459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:25.694741 27459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/data/instance:
uuid: "1b69ef6123274bf09f22200a335bbd9f"
format_stamp: "Formatted at 2026-09-28 08:53:25 on dist-test-slave-pbjp"
server_key: "af8aaa5fc515e12d3ffd356d6ac4d11a"
server_key_iv: "88fbdb4bdc7859134445413fd8602334"
server_key_version: "encryptionkey@0"
I20260928 08:53:25.695051 27459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance:
uuid: "1b69ef6123274bf09f22200a335bbd9f"
format_stamp: "Formatted at 2026-09-28 08:53:25 on dist-test-slave-pbjp"
server_key: "af8aaa5fc515e12d3ffd356d6ac4d11a"
server_key_iv: "88fbdb4bdc7859134445413fd8602334"
server_key_version: "encryptionkey@0"
I20260928 08:53:25.696272 27459 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 08:53:25.696955 27474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:25.697139 27459 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:53:25.697225 27459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "1b69ef6123274bf09f22200a335bbd9f"
format_stamp: "Formatted at 2026-09-28 08:53:25 on dist-test-slave-pbjp"
server_key: "af8aaa5fc515e12d3ffd356d6ac4d11a"
server_key_iv: "88fbdb4bdc7859134445413fd8602334"
server_key_version: "encryptionkey@0"
I20260928 08:53:25.697289 27459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:25.706234 27459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:25.708021 27459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:25.708154 27459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:25.708374 27459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:53:25.708667 27459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:25.708699 27459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:25.708727 27459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:25.708750 27459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:25.714655 27459 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:34917
I20260928 08:53:25.714691 27587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:34917 every 8 connection(s)
I20260928 08:53:25.715011 27459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
Sep 28 08:53:25 dist-test-slave-pbjp krb5kdc[27217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585605, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:25.723776 27301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:25.715691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:36139 (local address 127.23.167.254:34515)
0928 08:53:25.715737 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:25.715739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:25.715932 (+ 193us) server_negotiation.cc:407] Connection header received
0928 08:53:25.716472 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:25.716473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:25.716486 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:25.716539 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:25.717561 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:25.718061 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:25.719048 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:25.719239 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:25.719307 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:25.721378 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:25.721379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:25.721380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:25.721388 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:25.722509 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:25.722882 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:25.722883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:25.722883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:25.722940 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:25.723162 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:25.723163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:25.723163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:25.723277 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:25.723333 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:25.723599 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 08:53:25.723682 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 08:53:25.723970 27590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:25.715664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34515 (local address 127.23.167.194:36139)
0928 08:53:25.715814 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:25.715850 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:25.716351 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:25.716613 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:25.716618 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:25.717258 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:25.717460 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:25.717465 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:25.718146 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:25.718147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:25.718963 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:25.718965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:25.719321 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:25.719322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:25.719441 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:25.719903 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:25.719912 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:25.721287 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:25.722585 (+ 1298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:25.722594 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:25.722596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:25.722811 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:25.722995 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:25.722996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:25.722996 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:25.723106 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:25.723332 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:25.723338 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:25.723481 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 08:53:25.723570 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 08:53:25.723735 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20260928 08:53:25.724466 27588 heartbeater.cc:347] Connected to a master server at 127.23.167.254:34515
I20260928 08:53:25.724509 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 27459
I20260928 08:53:25.724562 27588 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:25.724598 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance
I20260928 08:53:25.724745 27588 heartbeater.cc:511] Master 127.23.167.254:34515 requested a full tablet report, sending...
I20260928 08:53:25.724779 24223 external_mini_cluster.cc:1428] Setting key 85a08075ef3fcb0715d71f4740eefb30
I20260928 08:53:25.725176 27261 ts_manager.cc:194] Registered new tserver with Master: 1b69ef6123274bf09f22200a335bbd9f (127.23.167.194:34917)
I20260928 08:53:25.725612 27261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.194@KRBTEST.COM'} at 127.23.167.194:36139
WARNING: no policy specified for kudu/127.23.167.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
I20260928 08:53:25.748055 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:0
--local_ip_for_outbound_sockets=127.23.167.195
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:34515
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:44569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:25.830418 27593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:25.830639 27593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:25.830672 27593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:25.831841 27593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:25.831884 27593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:25.831918 27593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:53:25.833410 27593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:44569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:34515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:25.833652 27593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:25.833865 27593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 08:53:25.839020 27593 server_base.cc:1034] running on GCE node
W20260928 08:53:25.839044 27599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:25.839063 27601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:25.839054 27598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:25.839429 27593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:25.839658 27593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:25.840806 27593 hybrid_clock.cc:648] HybridClock initialized: now 1790585605840790 us; error 29 us; skew 500 ppm
Sep 28 08:53:25 dist-test-slave-pbjp krb5kdc[27217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585605, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:25.843142 27593 init.cc:377] Logged in from keytab as kudu/127.23.167.195@KRBTEST.COM (short username kudu)
I20260928 08:53:25.843601 27593 webserver.cc:460] Webserver started at http://127.23.167.195:46587/ using document root <none> and password file <none>
I20260928 08:53:25.843818 27593 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:25.843868 27593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:25.843994 27593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:25.845010 27593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/data/instance:
uuid: "226ccf2db568482fa6950e1b8231c57c"
format_stamp: "Formatted at 2026-09-28 08:53:25 on dist-test-slave-pbjp"
server_key: "056255894bdcaaa7725554d3e959c120"
server_key_iv: "0da62984aee3c3affc2099ec4d54d8d8"
server_key_version: "encryptionkey@0"
I20260928 08:53:25.845414 27593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance:
uuid: "226ccf2db568482fa6950e1b8231c57c"
format_stamp: "Formatted at 2026-09-28 08:53:25 on dist-test-slave-pbjp"
server_key: "056255894bdcaaa7725554d3e959c120"
server_key_iv: "0da62984aee3c3affc2099ec4d54d8d8"
server_key_version: "encryptionkey@0"
I20260928 08:53:25.846637 27593 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:53:25.847250 27608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:25.847388 27593 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:53:25.847469 27593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "226ccf2db568482fa6950e1b8231c57c"
format_stamp: "Formatted at 2026-09-28 08:53:25 on dist-test-slave-pbjp"
server_key: "056255894bdcaaa7725554d3e959c120"
server_key_iv: "0da62984aee3c3affc2099ec4d54d8d8"
server_key_version: "encryptionkey@0"
I20260928 08:53:25.847536 27593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:25.864925 27593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:25.866818 27593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:25.866961 27593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:25.867182 27593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:53:25.867504 27593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:25.867543 27593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:25.867573 27593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:25.867590 27593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:25.873169 27593 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:41793
I20260928 08:53:25.873227 27721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:41793 every 8 connection(s)
I20260928 08:53:25.873528 27593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
Sep 28 08:53:25 dist-test-slave-pbjp krb5kdc[27217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585605, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:25.882200 27301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:25.874194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:47375 (local address 127.23.167.254:34515)
0928 08:53:25.874244 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:25.874247 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:25.874484 (+ 237us) server_negotiation.cc:407] Connection header received
0928 08:53:25.874950 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:25.874951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:25.874961 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:25.875010 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:25.875895 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:25.876379 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:25.877457 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:25.877652 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:25.877716 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:25.879737 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:25.879738 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:25.879738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:25.879745 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:25.880945 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:25.881315 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:25.881315 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:25.881316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:25.881363 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:25.881577 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:25.881577 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:25.881577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:25.881675 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:25.881730 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:25.882030 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 08:53:25.882107 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 08:53:25.882403 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:25.874169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34515 (local address 127.23.167.195:47375)
0928 08:53:25.874335 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:25.874402 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:25.874868 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:25.875068 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:25.875071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:25.875629 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:25.875828 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:25.875832 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:25.876446 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:25.876446 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:25.877297 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:25.877300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:25.877733 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:25.877734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:25.877838 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:25.878275 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:25.878283 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:25.879655 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:25.881015 (+ 1360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:25.881020 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:25.881022 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:25.881251 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:25.881435 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:25.881435 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:25.881436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:25.881528 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:25.881742 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:25.881749 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:25.881893 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 08:53:25.882001 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 08:53:25.882181 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
I20260928 08:53:25.882926 27722 heartbeater.cc:347] Connected to a master server at 127.23.167.254:34515
I20260928 08:53:25.882970 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 27593
I20260928 08:53:25.883034 27722 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:25.883050 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance
I20260928 08:53:25.883235 24223 external_mini_cluster.cc:1428] Setting key 2f487fa361f6808d587f7ef9c373eb0a
I20260928 08:53:25.883280 27722 heartbeater.cc:511] Master 127.23.167.254:34515 requested a full tablet report, sending...
I20260928 08:53:25.883692 27261 ts_manager.cc:194] Registered new tserver with Master: 226ccf2db568482fa6950e1b8231c57c (127.23.167.195:41793)
I20260928 08:53:25.884119 24223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:53:25.884160 27261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.195@KRBTEST.COM'} at 127.23.167.195:47375
Sep 28 08:53:25 dist-test-slave-pbjp krb5kdc[27217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585605, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 08:53:25 dist-test-slave-pbjp krb5kdc[27217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585605, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:25.905754 27301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:25.898818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35374 (local address 127.23.167.254:34515)
0928 08:53:25.898898 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:25.898900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:25.898955 (+ 55us) server_negotiation.cc:407] Connection header received
0928 08:53:25.899039 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:25.899040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:25.899050 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:25.899089 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:25.899662 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:25.900198 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:25.900946 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:25.901167 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:25.901212 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:25.903386 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:25.903387 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:25.903388 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:25.903396 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:25.904555 (+ 1159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:25.905084 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:25.905086 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:25.905086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:25.905156 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:25.905325 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:25.905325 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:25.905326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:25.905468 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:25.905540 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:25.905620 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 08:53:25.905684 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 08:53:25.906003 27261 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35374
I20260928 08:53:25.906386 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 27325
I20260928 08:53:25.910815 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 27459
I20260928 08:53:25.914633 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 27593
I20260928 08:53:25.918458 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 27230
2026-09-28T08:53:25Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1706 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:53:25 dist-test-slave-pbjp krb5kdc[27734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:53:25 dist-test-slave-pbjp krb5kdc[27734](info): set up 2 sockets
Sep 28 08:53:25 dist-test-slave-pbjp krb5kdc[27734](info): commencing operation
krb5kdc: starting...
W20260928 08:53:26.876611 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:53:26 dist-test-slave-pbjp krb5kdc[27734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:53:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:53:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.167.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
I20260928 08:53:26.988197 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:40413
--webserver_interface=127.23.167.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:45065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:40413
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:27.068485 27747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:27.068711 27747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:27.068743 27747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:27.070025 27747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:53:27.070070 27747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:27.070087 27747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:27.070104 27747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:53:27.070128 27747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:53:27.071631 27747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:45065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:40413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:40413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:27.071877 27747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:27.072093 27747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:27.076624 27755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:27.076696 27747 server_base.cc:1034] running on GCE node
W20260928 08:53:27.076630 27753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:27.076609 27752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:27.077054 27747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:27.077345 27747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:27.078500 27747 hybrid_clock.cc:648] HybridClock initialized: now 1790585607078486 us; error 29 us; skew 500 ppm
Sep 28 08:53:27 dist-test-slave-pbjp krb5kdc[27734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585607, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:27.086274 27747 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:53:27.086818 27747 webserver.cc:460] Webserver started at http://127.23.167.254:46519/ using document root <none> and password file <none>
I20260928 08:53:27.087020 27747 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:27.087070 27747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:27.087194 27747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:27.088259 27747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "a82064c6e16546e7988007aa4f533aa0"
format_stamp: "Formatted at 2026-09-28 08:53:27 on dist-test-slave-pbjp"
server_key: "86cdd48fd841e44a7c4aa628afcc8686"
server_key_iv: "af1826a266c457946a899dd92d578869"
server_key_version: "encryptionkey@0"
I20260928 08:53:27.088603 27747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "a82064c6e16546e7988007aa4f533aa0"
format_stamp: "Formatted at 2026-09-28 08:53:27 on dist-test-slave-pbjp"
server_key: "86cdd48fd841e44a7c4aa628afcc8686"
server_key_iv: "af1826a266c457946a899dd92d578869"
server_key_version: "encryptionkey@0"
I20260928 08:53:27.089936 27747 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 08:53:27.090677 27762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:27.090847 27747 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:53:27.090934 27747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "a82064c6e16546e7988007aa4f533aa0"
format_stamp: "Formatted at 2026-09-28 08:53:27 on dist-test-slave-pbjp"
server_key: "86cdd48fd841e44a7c4aa628afcc8686"
server_key_iv: "af1826a266c457946a899dd92d578869"
server_key_version: "encryptionkey@0"
I20260928 08:53:27.090994 27747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:27.098800 27747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:27.100807 27747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:27.100939 27747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:27.104691 27747 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:40413
I20260928 08:53:27.104733 27814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:40413 every 8 connection(s)
I20260928 08:53:27.105252 27747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:53:27.105964 27815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:27.108230 27815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a82064c6e16546e7988007aa4f533aa0: Bootstrap starting.
I20260928 08:53:27.108811 27815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a82064c6e16546e7988007aa4f533aa0: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:27.109127 27815 log.cc:824] T 00000000000000000000000000000000 P a82064c6e16546e7988007aa4f533aa0: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:27.109829 27815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a82064c6e16546e7988007aa4f533aa0: No bootstrap required, opened a new log
I20260928 08:53:27.111428 27815 raft_consensus.cc:348] T 00000000000000000000000000000000 P a82064c6e16546e7988007aa4f533aa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a82064c6e16546e7988007aa4f533aa0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 40413 } }
I20260928 08:53:27.111572 27815 raft_consensus.cc:374] T 00000000000000000000000000000000 P a82064c6e16546e7988007aa4f533aa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:27.111593 27815 raft_consensus.cc:729] T 00000000000000000000000000000000 P a82064c6e16546e7988007aa4f533aa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a82064c6e16546e7988007aa4f533aa0, State: Initialized, Role: FOLLOWER
I20260928 08:53:27.111734 27815 consensus_queue.cc:260] T 00000000000000000000000000000000 P a82064c6e16546e7988007aa4f533aa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a82064c6e16546e7988007aa4f533aa0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 40413 } }
I20260928 08:53:27.111801 27815 raft_consensus.cc:388] T 00000000000000000000000000000000 P a82064c6e16546e7988007aa4f533aa0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:53:27.111824 27815 raft_consensus.cc:482] T 00000000000000000000000000000000 P a82064c6e16546e7988007aa4f533aa0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:53:27.111856 27815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a82064c6e16546e7988007aa4f533aa0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:27.112589 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 27747
I20260928 08:53:27.112681 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:53:27.112763 27815 raft_consensus.cc:504] T 00000000000000000000000000000000 P a82064c6e16546e7988007aa4f533aa0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a82064c6e16546e7988007aa4f533aa0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 40413 } }
I20260928 08:53:27.112852 24223 external_mini_cluster.cc:1428] Setting key ace7fea5f26bce6056608c0285e6acac
I20260928 08:53:27.112870 27815 leader_election.cc:302] T 00000000000000000000000000000000 P a82064c6e16546e7988007aa4f533aa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a82064c6e16546e7988007aa4f533aa0; no voters:
I20260928 08:53:27.113040 27815 leader_election.cc:288] T 00000000000000000000000000000000 P a82064c6e16546e7988007aa4f533aa0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:53:27.113106 27818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a82064c6e16546e7988007aa4f533aa0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:53:27.113241 27818 raft_consensus.cc:686] T 00000000000000000000000000000000 P a82064c6e16546e7988007aa4f533aa0 [term 1 LEADER]: Becoming Leader. State: Replica: a82064c6e16546e7988007aa4f533aa0, State: Running, Role: LEADER
I20260928 08:53:27.113373 27815 sys_catalog.cc:564] T 00000000000000000000000000000000 P a82064c6e16546e7988007aa4f533aa0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:53:27.113384 27818 consensus_queue.cc:237] T 00000000000000000000000000000000 P a82064c6e16546e7988007aa4f533aa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a82064c6e16546e7988007aa4f533aa0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 40413 } }
I20260928 08:53:27.113790 27820 sys_catalog.cc:455] T 00000000000000000000000000000000 P a82064c6e16546e7988007aa4f533aa0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a82064c6e16546e7988007aa4f533aa0. Latest consensus state: current_term: 1 leader_uuid: "a82064c6e16546e7988007aa4f533aa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a82064c6e16546e7988007aa4f533aa0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 40413 } } }
I20260928 08:53:27.113874 27820 sys_catalog.cc:458] T 00000000000000000000000000000000 P a82064c6e16546e7988007aa4f533aa0 [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:27.114106 27819 sys_catalog.cc:455] T 00000000000000000000000000000000 P a82064c6e16546e7988007aa4f533aa0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a82064c6e16546e7988007aa4f533aa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a82064c6e16546e7988007aa4f533aa0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 40413 } } }
I20260928 08:53:27.114214 27819 sys_catalog.cc:458] T 00000000000000000000000000000000 P a82064c6e16546e7988007aa4f533aa0 [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:27.114624 27827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:53:27.115206 27827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:53:27.116827 27827 catalog_manager.cc:1347] Generated new cluster ID: cf8d375353b54582b50b248c18229b52
I20260928 08:53:27.116874 27827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 08:53:27 dist-test-slave-pbjp krb5kdc[27734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:27.124464 27822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:27.113343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38594 (local address 127.23.167.254:40413)
0928 08:53:27.113566 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:27.113615 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:27.113898 (+ 283us) server_negotiation.cc:407] Connection header received
0928 08:53:27.114141 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:27.114151 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:27.114228 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:27.114344 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:27.115301 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:27.116495 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:27.117554 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:27.118191 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:27.118302 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:27.120945 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:27.120949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:27.120949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:27.120963 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:27.122868 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:27.123232 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:27.123235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:27.123236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:27.123348 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:27.123528 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:27.123529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:27.123529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:27.123853 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:27.124022 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:27.124090 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 08:53:27.124231 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":61,"threads_started":1}
I20260928 08:53:27.130122 27827 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:53:27.130844 27827 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:53:27.148586 27827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a82064c6e16546e7988007aa4f533aa0: Generated new TSK 0
I20260928 08:53:27.148854 27827 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.167.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
I20260928 08:53:27.179622 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:0
--local_ip_for_outbound_sockets=127.23.167.193
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:40413
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:45065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:27.259917 27841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:27.260124 27841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:27.260147 27841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:27.261317 27841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:27.261363 27841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:27.261389 27841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:53:27.262880 27841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:45065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:40413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:27.263099 27841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:27.263306 27841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:27.268126 27847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:27.268121 27849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:27.268121 27846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:27.268146 27841 server_base.cc:1034] running on GCE node
I20260928 08:53:27.268528 27841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:27.268781 27841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:27.269920 27841 hybrid_clock.cc:648] HybridClock initialized: now 1790585607269909 us; error 41 us; skew 500 ppm
Sep 28 08:53:27 dist-test-slave-pbjp krb5kdc[27734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585607, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:27.272352 27841 init.cc:377] Logged in from keytab as kudu/127.23.167.193@KRBTEST.COM (short username kudu)
I20260928 08:53:27.272807 27841 webserver.cc:460] Webserver started at http://127.23.167.193:36091/ using document root <none> and password file <none>
I20260928 08:53:27.273010 27841 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:27.273075 27841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:27.273198 27841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:27.274220 27841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-0/data/instance:
uuid: "d0559908be3444a094684d7f5160f127"
format_stamp: "Formatted at 2026-09-28 08:53:27 on dist-test-slave-pbjp"
server_key: "75b6a0cca67b5c6c6c5e0018b1113bac"
server_key_iv: "8581c6ff793ccdc9be81abdfbfa3c64d"
server_key_version: "encryptionkey@0"
I20260928 08:53:27.274556 27841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance:
uuid: "d0559908be3444a094684d7f5160f127"
format_stamp: "Formatted at 2026-09-28 08:53:27 on dist-test-slave-pbjp"
server_key: "75b6a0cca67b5c6c6c5e0018b1113bac"
server_key_iv: "8581c6ff793ccdc9be81abdfbfa3c64d"
server_key_version: "encryptionkey@0"
I20260928 08:53:27.275801 27841 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:53:27.276504 27856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:27.276641 27841 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:53:27.276722 27841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "d0559908be3444a094684d7f5160f127"
format_stamp: "Formatted at 2026-09-28 08:53:27 on dist-test-slave-pbjp"
server_key: "75b6a0cca67b5c6c6c5e0018b1113bac"
server_key_iv: "8581c6ff793ccdc9be81abdfbfa3c64d"
server_key_version: "encryptionkey@0"
I20260928 08:53:27.276784 27841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:27.293268 27841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:27.295384 27841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:27.295531 27841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:27.295742 27841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:53:27.296051 27841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:27.296084 27841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:27.296114 27841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:27.296132 27841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:27.302563 27841 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:40313
I20260928 08:53:27.302636 27969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:40313 every 8 connection(s)
I20260928 08:53:27.303062 27841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
I20260928 08:53:27.304471 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 27841
I20260928 08:53:27.304581 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance
I20260928 08:53:27.304774 24223 external_mini_cluster.cc:1428] Setting key 5f9c8ae68c51764646742a329b3b1186
Sep 28 08:53:27 dist-test-slave-pbjp krb5kdc[27734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585607, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.167.194@KRBTEST.COM; defaulting to no policy
I20260928 08:53:27.313995 27822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:27.305112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:50449 (local address 127.23.167.254:40413)
0928 08:53:27.306233 (+ 1121us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:27.306235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:27.306247 (+ 12us) server_negotiation.cc:407] Connection header received
0928 08:53:27.306374 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:27.306375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:27.306389 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:27.306437 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:27.307088 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:27.307597 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:27.308630 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:27.308853 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:27.308909 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:27.311320 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:27.311322 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:27.311322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:27.311329 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:27.312653 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:27.313096 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:27.313097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:27.313097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:27.313157 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:27.313374 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:27.313375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:27.313375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:27.313509 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:27.313559 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:27.313813 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 08:53:27.313924 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1060}
I20260928 08:53:27.314141 27972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:27.304784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:40413 (local address 127.23.167.193:50449)
0928 08:53:27.304966 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:27.305048 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:27.306268 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:27.306514 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:27.306518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:27.306783 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:27.306983 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:27.306987 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:27.307682 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:27.307682 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:27.308518 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:27.308520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:27.308925 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:27.308925 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:27.309082 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:27.309617 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:27.309626 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:27.311224 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:27.312739 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:27.312746 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:27.312748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:27.312997 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:27.313214 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:27.313215 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:27.313215 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:27.313316 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:27.313557 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:27.313561 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:27.313681 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 08:53:27.313769 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 08:53:27.313946 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.194@KRBTEST.COM" created.
I20260928 08:53:27.314633 27970 heartbeater.cc:347] Connected to a master server at 127.23.167.254:40413
I20260928 08:53:27.314735 27970 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:27.314936 27970 heartbeater.cc:511] Master 127.23.167.254:40413 requested a full tablet report, sending...
I20260928 08:53:27.316187 27779 ts_manager.cc:194] Registered new tserver with Master: d0559908be3444a094684d7f5160f127 (127.23.167.193:40313)
I20260928 08:53:27.316785 27779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.193@KRBTEST.COM'} at 127.23.167.193:50449
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
I20260928 08:53:27.328043 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:0
--local_ip_for_outbound_sockets=127.23.167.194
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:40413
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:45065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:27.409143 27975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:27.409358 27975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:27.409380 27975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:27.410553 27975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:27.410600 27975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:27.410629 27975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:53:27.412182 27975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:45065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:40413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:27.412431 27975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:27.412645 27975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:27.417500 27983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:27.417524 27981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:27.417524 27980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:27.417740 27975 server_base.cc:1034] running on GCE node
I20260928 08:53:27.417963 27975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:27.418179 27975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:27.419344 27975 hybrid_clock.cc:648] HybridClock initialized: now 1790585607419326 us; error 32 us; skew 500 ppm
Sep 28 08:53:27 dist-test-slave-pbjp krb5kdc[27734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585607, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:27.421849 27975 init.cc:377] Logged in from keytab as kudu/127.23.167.194@KRBTEST.COM (short username kudu)
I20260928 08:53:27.422274 27975 webserver.cc:460] Webserver started at http://127.23.167.194:33931/ using document root <none> and password file <none>
I20260928 08:53:27.422472 27975 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:27.422513 27975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:27.422603 27975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:27.423595 27975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-1/data/instance:
uuid: "afd130fc21da4f2fa9455339a788d2f9"
format_stamp: "Formatted at 2026-09-28 08:53:27 on dist-test-slave-pbjp"
server_key: "4b5788b5d1a3554bdf98f41bb44b3a4e"
server_key_iv: "59c1816c2823eded56bd02199d3ba124"
server_key_version: "encryptionkey@0"
I20260928 08:53:27.423892 27975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance:
uuid: "afd130fc21da4f2fa9455339a788d2f9"
format_stamp: "Formatted at 2026-09-28 08:53:27 on dist-test-slave-pbjp"
server_key: "4b5788b5d1a3554bdf98f41bb44b3a4e"
server_key_iv: "59c1816c2823eded56bd02199d3ba124"
server_key_version: "encryptionkey@0"
I20260928 08:53:27.425136 27975 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 08:53:27.425808 27990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:27.425952 27975 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:53:27.426038 27975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "afd130fc21da4f2fa9455339a788d2f9"
format_stamp: "Formatted at 2026-09-28 08:53:27 on dist-test-slave-pbjp"
server_key: "4b5788b5d1a3554bdf98f41bb44b3a4e"
server_key_iv: "59c1816c2823eded56bd02199d3ba124"
server_key_version: "encryptionkey@0"
I20260928 08:53:27.426096 27975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:27.438123 27975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:27.440096 27975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:27.440205 27975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:27.440385 27975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:53:27.440641 27975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:27.440671 27975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:27.440693 27975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:27.440706 27975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:27.446115 27975 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:39883
I20260928 08:53:27.446182 28103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:39883 every 8 connection(s)
I20260928 08:53:27.446462 27975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
Sep 28 08:53:27 dist-test-slave-pbjp krb5kdc[27734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585607, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:27.452594 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 27975
I20260928 08:53:27.452679 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance
I20260928 08:53:27.452864 24223 external_mini_cluster.cc:1428] Setting key 617da29ffb897f61f5b2de319e611064
I20260928 08:53:27.456090 27822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:27.447111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:58543 (local address 127.23.167.254:40413)
0928 08:53:27.447155 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:27.447157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:27.447383 (+ 226us) server_negotiation.cc:407] Connection header received
0928 08:53:27.448323 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:27.448324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:27.448337 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:27.448390 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:27.449030 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:27.449586 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:27.450597 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:27.450797 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:27.450856 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:27.453180 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:27.453182 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:27.453182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:27.453190 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:27.454744 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:27.455162 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:27.455164 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:27.455164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:27.455225 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:27.455468 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:27.455469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:27.455469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:27.455587 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:27.455643 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:27.455907 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 08:53:27.456000 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 08:53:27.456367 28106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:27.447096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:40413 (local address 127.23.167.194:58543)
0928 08:53:27.447260 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:27.447296 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:27.448208 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:27.448475 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:27.448479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:27.448737 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:27.448932 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:27.448937 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:27.449675 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:27.449676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:27.450509 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:27.450512 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:27.450867 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:27.450868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:27.451005 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:27.451511 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:27.451520 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:27.453090 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:27.454841 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:27.454847 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:27.454849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:27.455075 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:27.455287 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:27.455288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:27.455289 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:27.455409 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:27.455639 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:27.455645 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:27.455789 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 08:53:27.455880 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 08:53:27.456040 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
I20260928 08:53:27.456791 28104 heartbeater.cc:347] Connected to a master server at 127.23.167.254:40413
I20260928 08:53:27.456887 28104 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:27.457056 28104 heartbeater.cc:511] Master 127.23.167.254:40413 requested a full tablet report, sending...
I20260928 08:53:27.457453 27779 ts_manager.cc:194] Registered new tserver with Master: afd130fc21da4f2fa9455339a788d2f9 (127.23.167.194:39883)
I20260928 08:53:27.457827 27779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.194@KRBTEST.COM'} at 127.23.167.194:58543
WARNING: no policy specified for kudu/127.23.167.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
I20260928 08:53:27.475891 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:0
--local_ip_for_outbound_sockets=127.23.167.195
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:40413
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:45065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:27.556387 28109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:27.556594 28109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:27.556615 28109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:27.557756 28109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:27.557791 28109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:27.557816 28109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:53:27.559242 28109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:45065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:40413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:27.559465 28109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:27.559643 28109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:27.564529 28115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:27.564529 28117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:27.564556 28109 server_base.cc:1034] running on GCE node
W20260928 08:53:27.564555 28114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:27.564935 28109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:27.565225 28109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:27.566371 28109 hybrid_clock.cc:648] HybridClock initialized: now 1790585607566355 us; error 34 us; skew 500 ppm
Sep 28 08:53:27 dist-test-slave-pbjp krb5kdc[27734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585607, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:27.568745 28109 init.cc:377] Logged in from keytab as kudu/127.23.167.195@KRBTEST.COM (short username kudu)
I20260928 08:53:27.569216 28109 webserver.cc:460] Webserver started at http://127.23.167.195:40179/ using document root <none> and password file <none>
I20260928 08:53:27.569432 28109 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:27.569480 28109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:27.569600 28109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:27.570647 28109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-2/data/instance:
uuid: "58657937ee61432f9be470c24b416f4d"
format_stamp: "Formatted at 2026-09-28 08:53:27 on dist-test-slave-pbjp"
server_key: "6e65eb2a5933c4d9ee12b01e2f75e77a"
server_key_iv: "fffdc072ffc8e36f353c80934ab9a0f3"
server_key_version: "encryptionkey@0"
I20260928 08:53:27.570984 28109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance:
uuid: "58657937ee61432f9be470c24b416f4d"
format_stamp: "Formatted at 2026-09-28 08:53:27 on dist-test-slave-pbjp"
server_key: "6e65eb2a5933c4d9ee12b01e2f75e77a"
server_key_iv: "fffdc072ffc8e36f353c80934ab9a0f3"
server_key_version: "encryptionkey@0"
I20260928 08:53:27.572175 28109 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:53:27.572921 28124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:27.573119 28109 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:53:27.573208 28109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "58657937ee61432f9be470c24b416f4d"
format_stamp: "Formatted at 2026-09-28 08:53:27 on dist-test-slave-pbjp"
server_key: "6e65eb2a5933c4d9ee12b01e2f75e77a"
server_key_iv: "fffdc072ffc8e36f353c80934ab9a0f3"
server_key_version: "encryptionkey@0"
I20260928 08:53:27.573274 28109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:27.588244 28109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:27.590067 28109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:27.590195 28109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:27.590422 28109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:53:27.590739 28109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:27.590772 28109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:27.590801 28109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:27.590826 28109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:27.596302 28109 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:35865
I20260928 08:53:27.596354 28237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:35865 every 8 connection(s)
I20260928 08:53:27.596807 28109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
I20260928 08:53:27.600605 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 28109
I20260928 08:53:27.600708 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance
I20260928 08:53:27.600898 24223 external_mini_cluster.cc:1428] Setting key 444fc1007319eef3c4389a34055fcd50
Sep 28 08:53:27 dist-test-slave-pbjp krb5kdc[27734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585607, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:27.605892 27822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:27.597511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:44939 (local address 127.23.167.254:40413)
0928 08:53:27.597581 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:27.597583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:27.597758 (+ 175us) server_negotiation.cc:407] Connection header received
0928 08:53:27.598645 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:27.598647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:27.598658 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:27.598712 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:27.599284 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:27.599791 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:27.601006 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:27.601301 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:27.601373 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:27.603523 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:27.603524 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:27.603525 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:27.603532 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:27.604762 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:27.605144 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:27.605144 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:27.605145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:27.605194 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:27.605393 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:27.605393 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:27.605394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:27.605501 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:27.605549 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:27.605766 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 08:53:27.605824 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 08:53:27.606106 28240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:27.597494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:40413 (local address 127.23.167.195:44939)
0928 08:53:27.597646 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:27.597681 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:27.598539 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:27.598770 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:27.598775 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:27.599013 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:27.599193 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:27.599197 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:27.599895 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:27.599896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:27.600927 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:27.600930 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:27.601444 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:27.601445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:27.601605 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:27.602095 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:27.602104 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:27.603436 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:27.604827 (+ 1391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:27.604831 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:27.604834 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:27.605058 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:27.605243 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:27.605244 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:27.605244 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:27.605348 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:27.605548 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:27.605552 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:27.605669 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 08:53:27.605751 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 08:53:27.605872 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260928 08:53:27.606534 28238 heartbeater.cc:347] Connected to a master server at 127.23.167.254:40413
I20260928 08:53:27.606624 28238 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:27.606796 28238 heartbeater.cc:511] Master 127.23.167.254:40413 requested a full tablet report, sending...
I20260928 08:53:27.607182 27779 ts_manager.cc:194] Registered new tserver with Master: 58657937ee61432f9be470c24b416f4d (127.23.167.195:35865)
I20260928 08:53:27.607542 27779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.195@KRBTEST.COM'} at 127.23.167.195:44939
I20260928 08:53:27.612627 24223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 08:53:27 dist-test-slave-pbjp krb5kdc[27734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 08:53:27 dist-test-slave-pbjp krb5kdc[27734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.193@KRBTEST.COM
I20260928 08:53:27.634150 28245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:27.627174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34790 (local address 127.23.167.193:40313)
0928 08:53:27.627327 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:27.627332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:27.627342 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:53:27.627369 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:27.627371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:27.627391 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:27.627460 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:27.627955 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:27.628733 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:27.629503 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:27.629844 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:27.629906 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:27.631909 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:27.631916 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:27.631917 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:27.631930 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:27.633261 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:27.633557 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:27.633559 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:27.633560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:27.633611 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:27.633740 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:27.633741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:27.633741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:27.633862 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:27.633975 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:27.634009 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 08:53:27.634068 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
Sep 28 08:53:27 dist-test-slave-pbjp krb5kdc[27734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:27.641206 27822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:27.635313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38608 (local address 127.23.167.254:40413)
0928 08:53:27.635375 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:27.635377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:27.635412 (+ 35us) server_negotiation.cc:407] Connection header received
0928 08:53:27.635470 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:27.635471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:27.635481 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:27.635523 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:27.635924 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:27.636388 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:27.637041 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:27.637292 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:27.637341 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:27.639282 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:27.639284 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:27.639284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:27.639292 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:27.640342 (+ 1050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:27.640604 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:27.640604 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:27.640605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:27.640648 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:27.640764 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:27.640764 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:27.640764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:27.640896 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:27.640962 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:27.641043 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 08:53:27.641139 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 08:53:27.641415 27779 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38608
I20260928 08:53:27.641717 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 27841
I20260928 08:53:27.646943 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 27975
I20260928 08:53:27.651046 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 28109
I20260928 08:53:27.654712 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 27747
2026-09-28T08:53:27Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1736 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:53:27 dist-test-slave-pbjp krb5kdc[28250](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:53:27 dist-test-slave-pbjp krb5kdc[28250](info): set up 2 sockets
Sep 28 08:53:27 dist-test-slave-pbjp krb5kdc[28250](info): commencing operation
krb5kdc: starting...
W20260928 08:53:28.608894 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:53:28 dist-test-slave-pbjp krb5kdc[28250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:53:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:53:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.167.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
I20260928 08:53:28.720871 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:33707
--webserver_interface=127.23.167.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:45149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:33707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:28.802572 28263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:28.802804 28263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:28.802829 28263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:28.803968 28263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:53:28.804005 28263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:28.804019 28263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:28.804030 28263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:53:28.804041 28263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:53:28.805627 28263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:45149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:33707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:33707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:28.805827 28263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:28.806000 28263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:28.810662 28271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:28.810796 28263 server_base.cc:1034] running on GCE node
W20260928 08:53:28.810664 28268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:28.810722 28269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:28.811195 28263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:28.811496 28263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:28.812649 28263 hybrid_clock.cc:648] HybridClock initialized: now 1790585608812634 us; error 30 us; skew 500 ppm
Sep 28 08:53:28 dist-test-slave-pbjp krb5kdc[28250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585608, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:28.815114 28263 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:53:28.815440 28263 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:28.815490 28263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:28.815585 28263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:28.816619 28263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "872ecf4a0a654ed6bfd50f8651eee89a"
format_stamp: "Formatted at 2026-09-28 08:53:28 on dist-test-slave-pbjp"
server_key: "e38a6df3a6e197e0df7836ad6b4a2849"
server_key_iv: "4fbdbe8182c76db92294d79120ab6073"
server_key_version: "encryptionkey@0"
I20260928 08:53:28.816926 28263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "872ecf4a0a654ed6bfd50f8651eee89a"
format_stamp: "Formatted at 2026-09-28 08:53:28 on dist-test-slave-pbjp"
server_key: "e38a6df3a6e197e0df7836ad6b4a2849"
server_key_iv: "4fbdbe8182c76db92294d79120ab6073"
server_key_version: "encryptionkey@0"
I20260928 08:53:28.818174 28263 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:53:28.818948 28277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:28.819123 28263 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:53:28.819211 28263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "872ecf4a0a654ed6bfd50f8651eee89a"
format_stamp: "Formatted at 2026-09-28 08:53:28 on dist-test-slave-pbjp"
server_key: "e38a6df3a6e197e0df7836ad6b4a2849"
server_key_iv: "4fbdbe8182c76db92294d79120ab6073"
server_key_version: "encryptionkey@0"
I20260928 08:53:28.819283 28263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:28.830679 28263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:28.832544 28263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:28.832664 28263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:28.836371 28263 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:33707
I20260928 08:53:28.836404 28329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:33707 every 8 connection(s)
I20260928 08:53:28.836681 28263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:53:28.837831 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:28.840149 28330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 872ecf4a0a654ed6bfd50f8651eee89a: Bootstrap starting.
I20260928 08:53:28.840723 28330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 872ecf4a0a654ed6bfd50f8651eee89a: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:28.840984 28330 log.cc:824] T 00000000000000000000000000000000 P 872ecf4a0a654ed6bfd50f8651eee89a: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:28.841642 28330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 872ecf4a0a654ed6bfd50f8651eee89a: No bootstrap required, opened a new log
I20260928 08:53:28.843329 28330 raft_consensus.cc:348] T 00000000000000000000000000000000 P 872ecf4a0a654ed6bfd50f8651eee89a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "872ecf4a0a654ed6bfd50f8651eee89a" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 33707 } }
I20260928 08:53:28.843494 28330 raft_consensus.cc:374] T 00000000000000000000000000000000 P 872ecf4a0a654ed6bfd50f8651eee89a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:28.843529 28330 raft_consensus.cc:729] T 00000000000000000000000000000000 P 872ecf4a0a654ed6bfd50f8651eee89a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 872ecf4a0a654ed6bfd50f8651eee89a, State: Initialized, Role: FOLLOWER
I20260928 08:53:28.843658 28330 consensus_queue.cc:260] T 00000000000000000000000000000000 P 872ecf4a0a654ed6bfd50f8651eee89a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "872ecf4a0a654ed6bfd50f8651eee89a" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 33707 } }
I20260928 08:53:28.843717 28330 raft_consensus.cc:388] T 00000000000000000000000000000000 P 872ecf4a0a654ed6bfd50f8651eee89a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:53:28.843746 28330 raft_consensus.cc:482] T 00000000000000000000000000000000 P 872ecf4a0a654ed6bfd50f8651eee89a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:53:28.843794 28330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 872ecf4a0a654ed6bfd50f8651eee89a [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:28.844641 28330 raft_consensus.cc:504] T 00000000000000000000000000000000 P 872ecf4a0a654ed6bfd50f8651eee89a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "872ecf4a0a654ed6bfd50f8651eee89a" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 33707 } }
I20260928 08:53:28.844741 28330 leader_election.cc:302] T 00000000000000000000000000000000 P 872ecf4a0a654ed6bfd50f8651eee89a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 872ecf4a0a654ed6bfd50f8651eee89a; no voters:
I20260928 08:53:28.844920 28330 leader_election.cc:288] T 00000000000000000000000000000000 P 872ecf4a0a654ed6bfd50f8651eee89a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:53:28.844970 28333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 872ecf4a0a654ed6bfd50f8651eee89a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:53:28.845101 28333 raft_consensus.cc:686] T 00000000000000000000000000000000 P 872ecf4a0a654ed6bfd50f8651eee89a [term 1 LEADER]: Becoming Leader. State: Replica: 872ecf4a0a654ed6bfd50f8651eee89a, State: Running, Role: LEADER
I20260928 08:53:28.845184 28333 consensus_queue.cc:237] T 00000000000000000000000000000000 P 872ecf4a0a654ed6bfd50f8651eee89a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "872ecf4a0a654ed6bfd50f8651eee89a" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 33707 } }
I20260928 08:53:28.845218 28330 sys_catalog.cc:564] T 00000000000000000000000000000000 P 872ecf4a0a654ed6bfd50f8651eee89a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:53:28.845597 28334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 872ecf4a0a654ed6bfd50f8651eee89a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "872ecf4a0a654ed6bfd50f8651eee89a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "872ecf4a0a654ed6bfd50f8651eee89a" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 33707 } } }
I20260928 08:53:28.845641 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 28263
I20260928 08:53:28.845724 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:53:28.845880 28334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 872ecf4a0a654ed6bfd50f8651eee89a [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:28.845921 24223 external_mini_cluster.cc:1428] Setting key c9a047d98ccbbdcaf5521c8741600263
I20260928 08:53:28.845817 28335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 872ecf4a0a654ed6bfd50f8651eee89a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 872ecf4a0a654ed6bfd50f8651eee89a. Latest consensus state: current_term: 1 leader_uuid: "872ecf4a0a654ed6bfd50f8651eee89a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "872ecf4a0a654ed6bfd50f8651eee89a" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 33707 } } }
I20260928 08:53:28.846170 28335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 872ecf4a0a654ed6bfd50f8651eee89a [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:28.846256 28343 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:53:28.846769 28343 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:53:28.848271 28343 catalog_manager.cc:1347] Generated new cluster ID: 8b945621215a43718bcc9198e50ca698
I20260928 08:53:28.848321 28343 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 08:53:28 dist-test-slave-pbjp krb5kdc[28250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:28.857656 28346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:28.846414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50252 (local address 127.23.167.254:33707)
0928 08:53:28.846882 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:28.846891 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:28.846925 (+ 34us) server_negotiation.cc:407] Connection header received
0928 08:53:28.847042 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:28.847050 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:28.847106 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:28.847252 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:28.847937 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:28.849344 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:28.850425 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:28.851033 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:28.851123 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:28.853971 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:28.853976 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:28.853976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:28.853990 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:28.855938 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:28.856320 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:28.856323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:28.856324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:28.856445 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:28.856623 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:28.856624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:28.856624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:28.856956 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:28.857163 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:28.857236 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 08:53:28.857401 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":85,"threads_started":1}
I20260928 08:53:28.860988 28343 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:53:28.861585 28343 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:53:28.864449 28343 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 872ecf4a0a654ed6bfd50f8651eee89a: Generated new TSK 0
I20260928 08:53:28.864647 28343 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.167.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
I20260928 08:53:28.887591 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:0
--local_ip_for_outbound_sockets=127.23.167.193
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:33707
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:45149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:28.967914 28356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:28.968118 28356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:28.968142 28356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:28.969350 28356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:28.969393 28356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:28.969419 28356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:53:28.971021 28356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:45149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:33707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:28.971247 28356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:28.971493 28356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:28.976238 28361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:28.976235 28362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:28.976254 28364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:28.976500 28356 server_base.cc:1034] running on GCE node
I20260928 08:53:28.976707 28356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:28.976933 28356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:28.978083 28356 hybrid_clock.cc:648] HybridClock initialized: now 1790585608978076 us; error 37 us; skew 500 ppm
Sep 28 08:53:28 dist-test-slave-pbjp krb5kdc[28250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585608, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:28.980505 28356 init.cc:377] Logged in from keytab as kudu/127.23.167.193@KRBTEST.COM (short username kudu)
I20260928 08:53:28.980851 28356 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:28.980901 28356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:28.981025 28356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:28.982051 28356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-0/data/instance:
uuid: "9bf6780949014442ac25f7697760cfa8"
format_stamp: "Formatted at 2026-09-28 08:53:28 on dist-test-slave-pbjp"
server_key: "7ff0ff5f1ee2f48d0d7ad69549606967"
server_key_iv: "0360058c2c4060b235d761eb4a29c0ea"
server_key_version: "encryptionkey@0"
I20260928 08:53:28.982362 28356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance:
uuid: "9bf6780949014442ac25f7697760cfa8"
format_stamp: "Formatted at 2026-09-28 08:53:28 on dist-test-slave-pbjp"
server_key: "7ff0ff5f1ee2f48d0d7ad69549606967"
server_key_iv: "0360058c2c4060b235d761eb4a29c0ea"
server_key_version: "encryptionkey@0"
I20260928 08:53:28.983520 28356 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 08:53:28.984164 28370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:28.984318 28356 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:53:28.984395 28356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "9bf6780949014442ac25f7697760cfa8"
format_stamp: "Formatted at 2026-09-28 08:53:28 on dist-test-slave-pbjp"
server_key: "7ff0ff5f1ee2f48d0d7ad69549606967"
server_key_iv: "0360058c2c4060b235d761eb4a29c0ea"
server_key_version: "encryptionkey@0"
I20260928 08:53:28.984457 28356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:29.003407 28356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:29.005177 28356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:29.005308 28356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:29.005512 28356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:53:29.005815 28356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:29.005851 28356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:29.005879 28356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:29.005899 28356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:29.011200 28356 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:45749
I20260928 08:53:29.011238 28483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:45749 every 8 connection(s)
I20260928 08:53:29.011480 28356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
I20260928 08:53:29.011880 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 28356
I20260928 08:53:29.011974 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance
I20260928 08:53:29.012161 24223 external_mini_cluster.cc:1428] Setting key 55dad57534c8dea72750fcbf634a434d
Sep 28 08:53:29 dist-test-slave-pbjp krb5kdc[28250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585608, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.167.194@KRBTEST.COM; defaulting to no policy
I20260928 08:53:29.021653 28346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:29.012195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:52965 (local address 127.23.167.254:33707)
0928 08:53:29.012247 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:29.012249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:29.012548 (+ 299us) server_negotiation.cc:407] Connection header received
0928 08:53:29.013164 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:29.013165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:29.013178 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:29.013241 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:29.014246 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.014786 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.016163 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.016401 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.016456 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:29.018967 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:29.018969 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:29.018969 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:29.018976 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:29.020394 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:29.020756 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:29.020757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:29.020757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:29.020804 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:29.021011 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:29.021012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:29.021012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:29.021156 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:29.021209 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:29.021494 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 08:53:29.021578 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 08:53:29.021762 28486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:29.012164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:33707 (local address 127.23.167.193:52965)
0928 08:53:29.012430 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:29.012469 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:29.013046 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:29.013300 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:29.013303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:29.013586 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:29.014151 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:29.014159 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.014880 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.014881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:29.016079 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:29.016083 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.016481 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.016482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:29.016642 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:29.017189 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:29.017201 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:29.018872 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:29.020469 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:29.020476 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:29.020479 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:29.020696 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:29.020860 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:29.020860 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:29.020860 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:29.020952 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:29.021207 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:29.021212 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:29.021350 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 08:53:29.021435 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 08:53:29.021583 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
I20260928 08:53:29.022186 28484 heartbeater.cc:347] Connected to a master server at 127.23.167.254:33707
I20260928 08:53:29.022276 28484 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:29.022409 28484 heartbeater.cc:511] Master 127.23.167.254:33707 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.194@KRBTEST.COM" created.
I20260928 08:53:29.022956 28294 ts_manager.cc:194] Registered new tserver with Master: 9bf6780949014442ac25f7697760cfa8 (127.23.167.193:45749)
I20260928 08:53:29.023578 28294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.193@KRBTEST.COM'} at 127.23.167.193:52965
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
I20260928 08:53:29.036990 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:0
--local_ip_for_outbound_sockets=127.23.167.194
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:33707
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:45149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:29.115196 28489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:29.115401 28489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:29.115432 28489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:29.116562 28489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:29.116598 28489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:29.116633 28489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:53:29.118098 28489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:45149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:33707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:29.118324 28489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:29.118527 28489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:29.123237 28497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:29.123225 28495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:29.123301 28489 server_base.cc:1034] running on GCE node
W20260928 08:53:29.123239 28494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:29.123584 28489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:29.123795 28489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:29.124959 28489 hybrid_clock.cc:648] HybridClock initialized: now 1790585609124943 us; error 26 us; skew 500 ppm
Sep 28 08:53:29 dist-test-slave-pbjp krb5kdc[28250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585609, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:29.127348 28489 init.cc:377] Logged in from keytab as kudu/127.23.167.194@KRBTEST.COM (short username kudu)
I20260928 08:53:29.127626 28489 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:29.127666 28489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:29.127758 28489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:29.128713 28489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-1/data/instance:
uuid: "b4c32366ec2f49d9a4f57daf07e375b9"
format_stamp: "Formatted at 2026-09-28 08:53:29 on dist-test-slave-pbjp"
server_key: "7cdea106da8fb9d4937cdf2117699e8c"
server_key_iv: "69f59ee0c129595ecbf9f0ba70c8c61f"
server_key_version: "encryptionkey@0"
I20260928 08:53:29.129006 28489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance:
uuid: "b4c32366ec2f49d9a4f57daf07e375b9"
format_stamp: "Formatted at 2026-09-28 08:53:29 on dist-test-slave-pbjp"
server_key: "7cdea106da8fb9d4937cdf2117699e8c"
server_key_iv: "69f59ee0c129595ecbf9f0ba70c8c61f"
server_key_version: "encryptionkey@0"
I20260928 08:53:29.130163 28489 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:53:29.130766 28503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:29.130919 28489 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:53:29.131013 28489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "b4c32366ec2f49d9a4f57daf07e375b9"
format_stamp: "Formatted at 2026-09-28 08:53:29 on dist-test-slave-pbjp"
server_key: "7cdea106da8fb9d4937cdf2117699e8c"
server_key_iv: "69f59ee0c129595ecbf9f0ba70c8c61f"
server_key_version: "encryptionkey@0"
I20260928 08:53:29.131069 28489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:29.154438 28489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:29.156381 28489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:29.156503 28489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:29.156747 28489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:53:29.157099 28489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:29.157135 28489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:29.157157 28489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:29.157181 28489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:29.162711 28489 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:43695
I20260928 08:53:29.162782 28616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:43695 every 8 connection(s)
I20260928 08:53:29.163026 28489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
Sep 28 08:53:29 dist-test-slave-pbjp krb5kdc[28250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585609, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:29.171886 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 28489
I20260928 08:53:29.171975 28346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:29.163734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:45229 (local address 127.23.167.254:33707)
0928 08:53:29.163799 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:29.163802 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:29.164014 (+ 212us) server_negotiation.cc:407] Connection header received
0928 08:53:29.164490 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:29.164492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:29.164502 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:29.164548 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:29.165519 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.166034 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.167027 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.167222 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.167267 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:29.169462 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:29.169464 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:29.169464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:29.169472 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:29.170831 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:29.171162 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:29.171162 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:29.171163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:29.171210 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:29.171390 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:29.171390 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:29.171391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:29.171495 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:29.171540 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:29.171775 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 08:53:29.171887 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 08:53:29.171998 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance
I20260928 08:53:29.172094 28619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:29.163710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:33707 (local address 127.23.167.194:45229)
0928 08:53:29.163892 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:29.163930 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:29.164408 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:29.164610 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:29.164613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:29.164875 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:29.165440 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:29.165447 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.166104 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.166105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:29.166951 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:29.166954 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.167281 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.167281 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:29.167390 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:29.167915 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:29.167924 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:29.169344 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:29.170898 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:29.170903 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:29.170904 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:29.171109 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:29.171254 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:29.171254 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:29.171255 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:29.171349 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:29.171537 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:29.171540 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:29.171665 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 08:53:29.171744 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 08:53:29.171871 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":90,"threads_started":1}
I20260928 08:53:29.172192 24223 external_mini_cluster.cc:1428] Setting key 56f48b2cf0a593feb956f50b3d43b4a6
I20260928 08:53:29.172592 28617 heartbeater.cc:347] Connected to a master server at 127.23.167.254:33707
I20260928 08:53:29.172699 28617 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:29.172884 28617 heartbeater.cc:511] Master 127.23.167.254:33707 requested a full tablet report, sending...
I20260928 08:53:29.173336 28294 ts_manager.cc:194] Registered new tserver with Master: b4c32366ec2f49d9a4f57daf07e375b9 (127.23.167.194:43695)
I20260928 08:53:29.173709 28294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.194@KRBTEST.COM'} at 127.23.167.194:45229
WARNING: no policy specified for kudu/127.23.167.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
I20260928 08:53:29.195565 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:0
--local_ip_for_outbound_sockets=127.23.167.195
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:33707
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:45149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:29.276394 28622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:29.276595 28622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:29.276618 28622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:29.277801 28622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:29.277843 28622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:29.277869 28622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:53:29.279404 28622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:45149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:33707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:29.279628 28622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:29.279844 28622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:29.284564 28627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:29.284617 28630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:29.284605 28622 server_base.cc:1034] running on GCE node
W20260928 08:53:29.284574 28628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:29.285017 28622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:29.285275 28622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:29.286425 28622 hybrid_clock.cc:648] HybridClock initialized: now 1790585609286408 us; error 29 us; skew 500 ppm
Sep 28 08:53:29 dist-test-slave-pbjp krb5kdc[28250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585609, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:29.288961 28622 init.cc:377] Logged in from keytab as kudu/127.23.167.195@KRBTEST.COM (short username kudu)
I20260928 08:53:29.289350 28622 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:29.289407 28622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:29.289554 28622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:29.290592 28622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-2/data/instance:
uuid: "6e3b64b877e746ed97cc9c057006200f"
format_stamp: "Formatted at 2026-09-28 08:53:29 on dist-test-slave-pbjp"
server_key: "9b74b8a23185829d44f4161413a7bcfe"
server_key_iv: "5865602ae09fdd3b5b0afaaf1d072836"
server_key_version: "encryptionkey@0"
I20260928 08:53:29.290920 28622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance:
uuid: "6e3b64b877e746ed97cc9c057006200f"
format_stamp: "Formatted at 2026-09-28 08:53:29 on dist-test-slave-pbjp"
server_key: "9b74b8a23185829d44f4161413a7bcfe"
server_key_iv: "5865602ae09fdd3b5b0afaaf1d072836"
server_key_version: "encryptionkey@0"
I20260928 08:53:29.292137 28622 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 08:53:29.292912 28636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:29.293097 28622 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:53:29.293187 28622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "6e3b64b877e746ed97cc9c057006200f"
format_stamp: "Formatted at 2026-09-28 08:53:29 on dist-test-slave-pbjp"
server_key: "9b74b8a23185829d44f4161413a7bcfe"
server_key_iv: "5865602ae09fdd3b5b0afaaf1d072836"
server_key_version: "encryptionkey@0"
I20260928 08:53:29.293253 28622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:29.328433 28622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:29.330593 28622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:29.330740 28622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:29.330968 28622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:53:29.331292 28622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:29.331327 28622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:29.331357 28622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:29.331373 28622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:29.337241 28622 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:35027
I20260928 08:53:29.337291 28749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:35027 every 8 connection(s)
I20260928 08:53:29.337577 28622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
I20260928 08:53:29.340479 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 28622
I20260928 08:53:29.340575 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance
I20260928 08:53:29.340759 24223 external_mini_cluster.cc:1428] Setting key b15e92881bafa8b76ede3c3e398d96d4
Sep 28 08:53:29 dist-test-slave-pbjp krb5kdc[28250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585609, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:29.346482 28346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:29.338321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:47625 (local address 127.23.167.254:33707)
0928 08:53:29.338400 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:29.338402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:29.338539 (+ 137us) server_negotiation.cc:407] Connection header received
0928 08:53:29.339034 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:29.339036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:29.339047 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:29.339104 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:29.340152 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.340757 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.341932 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.342125 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.342177 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:29.344165 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:29.344166 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:29.344166 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:29.344172 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:29.345314 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:29.345736 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:29.345737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:29.345737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:29.345794 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:29.345978 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:29.345978 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:29.345978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:29.346086 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:29.346132 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:29.346353 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 08:53:29.346417 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 08:53:29.346685 28752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:29.338242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:33707 (local address 127.23.167.195:47625)
0928 08:53:29.338417 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:29.338457 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:29.338945 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:29.339164 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:29.339167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:29.339513 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:29.340041 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:29.340048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.340853 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.340854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:29.341861 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:29.341864 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.342182 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.342183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:29.342283 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:29.342798 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:29.342806 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:29.344095 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:29.345448 (+ 1353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:29.345453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:29.345455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:29.345662 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:29.345833 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:29.345834 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:29.345834 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:29.345933 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:29.346127 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:29.346130 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:29.346254 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 08:53:29.346329 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 08:53:29.346476 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":66,"threads_started":1}
I20260928 08:53:29.347159 28750 heartbeater.cc:347] Connected to a master server at 127.23.167.254:33707
I20260928 08:53:29.347247 28750 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:29.347388 28750 heartbeater.cc:511] Master 127.23.167.254:33707 requested a full tablet report, sending...
I20260928 08:53:29.347724 28294 ts_manager.cc:194] Registered new tserver with Master: 6e3b64b877e746ed97cc9c057006200f (127.23.167.195:35027)
I20260928 08:53:29.348059 28294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.195@KRBTEST.COM'} at 127.23.167.195:47625
I20260928 08:53:29.352365 24223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:53:29.359200 28346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:29.353678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50262 (local address 127.23.167.254:33707)
0928 08:53:29.353723 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:29.353725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:29.353786 (+ 61us) server_negotiation.cc:407] Connection header received
0928 08:53:29.353831 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:29.353832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:29.353841 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:29.353882 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:29.354402 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.354862 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.355591 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.355761 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.355801 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:29.356951 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:29.356952 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:29.356952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:29.356956 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:29.358251 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:29.358574 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:29.358575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:29.358575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:29.358623 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:29.358739 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:29.358739 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:29.358739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:29.358866 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:29.358944 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:29.359079 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 08:53:29.359136 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 08:53:29.360878 28294 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:53:29.362488 28294 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:53:29.368577 28759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:29.364610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:43695 (local address 127.0.0.1:53200)
0928 08:53:29.364789 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:29.364828 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:29.364896 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:29.365347 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:29.365351 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:29.365353 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:53:29.365453 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:29.365456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.366362 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.366363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:29.367481 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:29.367484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.368289 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.368290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:29.368365 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:29.368377 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 08:53:29.368442 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 08:53:29.368497 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":20,"thread_start_us":77,"threads_started":1}
I20260928 08:53:29.368692 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:29.364869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42892 (local address 127.23.167.195:35027)
0928 08:53:29.365023 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:29.365029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:29.365043 (+ 14us) server_negotiation.cc:407] Connection header received
0928 08:53:29.365179 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:29.365183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:29.365203 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:29.365258 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:53:29.365520 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.366635 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.367750 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.368350 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.368424 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:29.368473 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:29.368571 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 08:53:29.368610 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":95,"threads_started":1}
I20260928 08:53:29.368664 28758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:29.364613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:35027 (local address 127.0.0.1:42892)
0928 08:53:29.364991 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:29.365011 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:29.365048 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:29.365368 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:29.365369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:29.365371 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:53:29.365442 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:29.365444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.366757 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.366757 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:29.367678 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:29.367681 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.368429 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.368430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:29.368501 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:29.368504 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:53:29.368548 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 08:53:29.368578 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":40,"thread_start_us":64,"threads_started":1}
I20260928 08:53:29.368989 28761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:29.364793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53200 (local address 127.23.167.194:43695)
0928 08:53:29.365101 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:29.365104 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:29.365114 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:53:29.365154 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:29.365156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:29.365172 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:29.365251 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:53:29.365530 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.366280 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.367575 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.368220 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.368281 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:29.368337 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:29.368880 (+ 543us) server_negotiation.cc:299] Negotiation successful
0928 08:53:29.368929 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":78,"threads_started":1}
I20260928 08:53:29.369458 28760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:29.364613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:45749 (local address 127.0.0.1:45464)
0928 08:53:29.365176 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:29.365185 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:29.365240 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:29.365596 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:29.365597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:29.365599 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:53:29.365632 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:29.365633 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.366831 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.366832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:29.367927 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:29.367930 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.369229 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.369230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:29.369290 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:29.369295 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 08:53:29.369349 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 08:53:29.369392 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":1,"thread_start_us":64,"threads_started":1}
I20260928 08:53:29.369495 28763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:29.364989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45464 (local address 127.23.167.193:45749)
0928 08:53:29.365169 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:29.365173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:29.365229 (+ 56us) server_negotiation.cc:407] Connection header received
0928 08:53:29.365302 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:29.365304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:29.365317 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:29.365499 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:53:29.365695 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.366737 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.368013 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.369039 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.369150 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:29.369205 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:29.369375 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 08:53:29.369420 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":102,"threads_started":1}
I20260928 08:53:29.369441 28684 tablet_service.cc:1467] Processing CreateTablet for tablet c4d09a3d9f9745a0be55687a6a264760 (DEFAULT_TABLE table=test-table [id=cd896c7708154f10aa95eec5df5c482a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:53:29.369629 28551 tablet_service.cc:1467] Processing CreateTablet for tablet c4d09a3d9f9745a0be55687a6a264760 (DEFAULT_TABLE table=test-table [id=cd896c7708154f10aa95eec5df5c482a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:53:29.369827 28684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4d09a3d9f9745a0be55687a6a264760. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:29.369922 28551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4d09a3d9f9745a0be55687a6a264760. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:29.370101 28418 tablet_service.cc:1467] Processing CreateTablet for tablet c4d09a3d9f9745a0be55687a6a264760 (DEFAULT_TABLE table=test-table [id=cd896c7708154f10aa95eec5df5c482a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:53:29.370388 28418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4d09a3d9f9745a0be55687a6a264760. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:29.372200 28764 tablet_bootstrap.cc:492] T c4d09a3d9f9745a0be55687a6a264760 P b4c32366ec2f49d9a4f57daf07e375b9: Bootstrap starting.
I20260928 08:53:29.372799 28764 tablet_bootstrap.cc:654] T c4d09a3d9f9745a0be55687a6a264760 P b4c32366ec2f49d9a4f57daf07e375b9: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:29.373008 28765 tablet_bootstrap.cc:492] T c4d09a3d9f9745a0be55687a6a264760 P 6e3b64b877e746ed97cc9c057006200f: Bootstrap starting.
I20260928 08:53:29.373394 28764 log.cc:824] T c4d09a3d9f9745a0be55687a6a264760 P b4c32366ec2f49d9a4f57daf07e375b9: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:29.373701 28765 tablet_bootstrap.cc:654] T c4d09a3d9f9745a0be55687a6a264760 P 6e3b64b877e746ed97cc9c057006200f: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:29.373733 28766 tablet_bootstrap.cc:492] T c4d09a3d9f9745a0be55687a6a264760 P 9bf6780949014442ac25f7697760cfa8: Bootstrap starting.
I20260928 08:53:29.374143 28764 tablet_bootstrap.cc:492] T c4d09a3d9f9745a0be55687a6a264760 P b4c32366ec2f49d9a4f57daf07e375b9: No bootstrap required, opened a new log
I20260928 08:53:29.374166 28765 log.cc:824] T c4d09a3d9f9745a0be55687a6a264760 P 6e3b64b877e746ed97cc9c057006200f: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:29.374217 28764 ts_tablet_manager.cc:1397] T c4d09a3d9f9745a0be55687a6a264760 P b4c32366ec2f49d9a4f57daf07e375b9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 08:53:29.374326 28766 tablet_bootstrap.cc:654] T c4d09a3d9f9745a0be55687a6a264760 P 9bf6780949014442ac25f7697760cfa8: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:29.374759 28765 tablet_bootstrap.cc:492] T c4d09a3d9f9745a0be55687a6a264760 P 6e3b64b877e746ed97cc9c057006200f: No bootstrap required, opened a new log
I20260928 08:53:29.374816 28765 ts_tablet_manager.cc:1397] T c4d09a3d9f9745a0be55687a6a264760 P 6e3b64b877e746ed97cc9c057006200f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:53:29.374871 28766 log.cc:824] T c4d09a3d9f9745a0be55687a6a264760 P 9bf6780949014442ac25f7697760cfa8: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:29.375610 28766 tablet_bootstrap.cc:492] T c4d09a3d9f9745a0be55687a6a264760 P 9bf6780949014442ac25f7697760cfa8: No bootstrap required, opened a new log
I20260928 08:53:29.375679 28766 ts_tablet_manager.cc:1397] T c4d09a3d9f9745a0be55687a6a264760 P 9bf6780949014442ac25f7697760cfa8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:53:29.376200 28764 raft_consensus.cc:348] T c4d09a3d9f9745a0be55687a6a264760 P b4c32366ec2f49d9a4f57daf07e375b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c32366ec2f49d9a4f57daf07e375b9" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 43695 } } peers { permanent_uuid: "9bf6780949014442ac25f7697760cfa8" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 45749 } } peers { permanent_uuid: "6e3b64b877e746ed97cc9c057006200f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 35027 } }
I20260928 08:53:29.376372 28764 raft_consensus.cc:374] T c4d09a3d9f9745a0be55687a6a264760 P b4c32366ec2f49d9a4f57daf07e375b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:29.376407 28764 raft_consensus.cc:729] T c4d09a3d9f9745a0be55687a6a264760 P b4c32366ec2f49d9a4f57daf07e375b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4c32366ec2f49d9a4f57daf07e375b9, State: Initialized, Role: FOLLOWER
I20260928 08:53:29.376389 28765 raft_consensus.cc:348] T c4d09a3d9f9745a0be55687a6a264760 P 6e3b64b877e746ed97cc9c057006200f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c32366ec2f49d9a4f57daf07e375b9" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 43695 } } peers { permanent_uuid: "9bf6780949014442ac25f7697760cfa8" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 45749 } } peers { permanent_uuid: "6e3b64b877e746ed97cc9c057006200f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 35027 } }
I20260928 08:53:29.376537 28764 consensus_queue.cc:260] T c4d09a3d9f9745a0be55687a6a264760 P b4c32366ec2f49d9a4f57daf07e375b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c32366ec2f49d9a4f57daf07e375b9" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 43695 } } peers { permanent_uuid: "9bf6780949014442ac25f7697760cfa8" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 45749 } } peers { permanent_uuid: "6e3b64b877e746ed97cc9c057006200f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 35027 } }
I20260928 08:53:29.376593 28765 raft_consensus.cc:374] T c4d09a3d9f9745a0be55687a6a264760 P 6e3b64b877e746ed97cc9c057006200f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:29.376634 28765 raft_consensus.cc:729] T c4d09a3d9f9745a0be55687a6a264760 P 6e3b64b877e746ed97cc9c057006200f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e3b64b877e746ed97cc9c057006200f, State: Initialized, Role: FOLLOWER
I20260928 08:53:29.376765 28765 consensus_queue.cc:260] T c4d09a3d9f9745a0be55687a6a264760 P 6e3b64b877e746ed97cc9c057006200f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c32366ec2f49d9a4f57daf07e375b9" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 43695 } } peers { permanent_uuid: "9bf6780949014442ac25f7697760cfa8" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 45749 } } peers { permanent_uuid: "6e3b64b877e746ed97cc9c057006200f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 35027 } }
I20260928 08:53:29.376797 28764 ts_tablet_manager.cc:1428] T c4d09a3d9f9745a0be55687a6a264760 P b4c32366ec2f49d9a4f57daf07e375b9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 08:53:29.376911 28617 heartbeater.cc:503] Master 127.23.167.254:33707 was elected leader, sending a full tablet report...
I20260928 08:53:29.377022 28765 ts_tablet_manager.cc:1428] T c4d09a3d9f9745a0be55687a6a264760 P 6e3b64b877e746ed97cc9c057006200f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:53:29.377036 28750 heartbeater.cc:503] Master 127.23.167.254:33707 was elected leader, sending a full tablet report...
I20260928 08:53:29.377864 28766 raft_consensus.cc:348] T c4d09a3d9f9745a0be55687a6a264760 P 9bf6780949014442ac25f7697760cfa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c32366ec2f49d9a4f57daf07e375b9" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 43695 } } peers { permanent_uuid: "9bf6780949014442ac25f7697760cfa8" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 45749 } } peers { permanent_uuid: "6e3b64b877e746ed97cc9c057006200f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 35027 } }
I20260928 08:53:29.378028 28766 raft_consensus.cc:374] T c4d09a3d9f9745a0be55687a6a264760 P 9bf6780949014442ac25f7697760cfa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:29.378072 28766 raft_consensus.cc:729] T c4d09a3d9f9745a0be55687a6a264760 P 9bf6780949014442ac25f7697760cfa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bf6780949014442ac25f7697760cfa8, State: Initialized, Role: FOLLOWER
I20260928 08:53:29.378185 28766 consensus_queue.cc:260] T c4d09a3d9f9745a0be55687a6a264760 P 9bf6780949014442ac25f7697760cfa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c32366ec2f49d9a4f57daf07e375b9" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 43695 } } peers { permanent_uuid: "9bf6780949014442ac25f7697760cfa8" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 45749 } } peers { permanent_uuid: "6e3b64b877e746ed97cc9c057006200f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 35027 } }
I20260928 08:53:29.378418 28766 ts_tablet_manager.cc:1428] T c4d09a3d9f9745a0be55687a6a264760 P 9bf6780949014442ac25f7697760cfa8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:53:29.378468 28484 heartbeater.cc:503] Master 127.23.167.254:33707 was elected leader, sending a full tablet report...
W20260928 08:53:29.413511 28618 tablet.cc:2369] T c4d09a3d9f9745a0be55687a6a264760 P b4c32366ec2f49d9a4f57daf07e375b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:53:29.512188 28485 tablet.cc:2369] T c4d09a3d9f9745a0be55687a6a264760 P 9bf6780949014442ac25f7697760cfa8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:53:29.554795 28772 raft_consensus.cc:482] T c4d09a3d9f9745a0be55687a6a264760 P 9bf6780949014442ac25f7697760cfa8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:53:29.554945 28772 raft_consensus.cc:504] T c4d09a3d9f9745a0be55687a6a264760 P 9bf6780949014442ac25f7697760cfa8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c32366ec2f49d9a4f57daf07e375b9" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 43695 } } peers { permanent_uuid: "9bf6780949014442ac25f7697760cfa8" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 45749 } } peers { permanent_uuid: "6e3b64b877e746ed97cc9c057006200f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 35027 } }
I20260928 08:53:29.555277 28772 leader_election.cc:288] T c4d09a3d9f9745a0be55687a6a264760 P 9bf6780949014442ac25f7697760cfa8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4c32366ec2f49d9a4f57daf07e375b9 (127.23.167.194:43695), 6e3b64b877e746ed97cc9c057006200f (127.23.167.195:35027)
I20260928 08:53:29.558095 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:29.555609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:34117 (local address 127.23.167.195:35027)
0928 08:53:29.555679 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:29.555681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:29.555689 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:53:29.555713 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:29.555713 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:29.555720 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:29.555764 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:53:29.555905 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.556369 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.557403 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.557775 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.557841 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:29.557850 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:29.557987 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 08:53:29.558030 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 08:53:29.558116 28761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:29.555514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:37313 (local address 127.23.167.194:43695)
0928 08:53:29.555569 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:29.555572 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:29.555716 (+ 144us) server_negotiation.cc:407] Connection header received
0928 08:53:29.555764 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:29.555765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:29.555775 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:29.555824 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:53:29.555964 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.556407 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.557437 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.557799 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.557867 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:29.557874 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:29.558015 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 08:53:29.558053 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 08:53:29.558145 28773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:29.555445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:35027 (local address 127.23.167.193:34117)
0928 08:53:29.555573 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:29.555583 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:29.555631 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:29.555775 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:29.555776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:29.555778 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:53:29.555851 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:29.555853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.556442 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.556443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:29.557318 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:29.557321 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.557853 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.557854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:29.557922 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:29.557926 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:53:29.557972 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 08:53:29.558015 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":41,"thread_start_us":75,"threads_started":1}
I20260928 08:53:29.558305 28774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:29.555444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:43695 (local address 127.23.167.193:37313)
0928 08:53:29.555628 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:29.555638 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:29.555714 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:29.555837 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:29.555838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:29.555840 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:53:29.555908 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:29.555910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.556481 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.556482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:29.557319 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:29.557321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.557874 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.557875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:29.557952 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:29.557955 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:53:29.558000 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 08:53:29.558039 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":6,"thread_start_us":59,"threads_started":1}
I20260928 08:53:29.558476 28704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4d09a3d9f9745a0be55687a6a264760" candidate_uuid: "9bf6780949014442ac25f7697760cfa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e3b64b877e746ed97cc9c057006200f" is_pre_election: true
I20260928 08:53:29.558648 28704 raft_consensus.cc:2454] T c4d09a3d9f9745a0be55687a6a264760 P 6e3b64b877e746ed97cc9c057006200f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bf6780949014442ac25f7697760cfa8 in term 0.
I20260928 08:53:29.558665 28571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4d09a3d9f9745a0be55687a6a264760" candidate_uuid: "9bf6780949014442ac25f7697760cfa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4c32366ec2f49d9a4f57daf07e375b9" is_pre_election: true
I20260928 08:53:29.558801 28571 raft_consensus.cc:2454] T c4d09a3d9f9745a0be55687a6a264760 P b4c32366ec2f49d9a4f57daf07e375b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bf6780949014442ac25f7697760cfa8 in term 0.
I20260928 08:53:29.558872 28371 leader_election.cc:302] T c4d09a3d9f9745a0be55687a6a264760 P 9bf6780949014442ac25f7697760cfa8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e3b64b877e746ed97cc9c057006200f, 9bf6780949014442ac25f7697760cfa8; no voters:
I20260928 08:53:29.558996 28772 raft_consensus.cc:2781] T c4d09a3d9f9745a0be55687a6a264760 P 9bf6780949014442ac25f7697760cfa8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:53:29.559049 28772 raft_consensus.cc:482] T c4d09a3d9f9745a0be55687a6a264760 P 9bf6780949014442ac25f7697760cfa8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:53:29.559063 28772 raft_consensus.cc:3037] T c4d09a3d9f9745a0be55687a6a264760 P 9bf6780949014442ac25f7697760cfa8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:29.559695 28772 raft_consensus.cc:504] T c4d09a3d9f9745a0be55687a6a264760 P 9bf6780949014442ac25f7697760cfa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c32366ec2f49d9a4f57daf07e375b9" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 43695 } } peers { permanent_uuid: "9bf6780949014442ac25f7697760cfa8" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 45749 } } peers { permanent_uuid: "6e3b64b877e746ed97cc9c057006200f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 35027 } }
I20260928 08:53:29.559806 28772 leader_election.cc:288] T c4d09a3d9f9745a0be55687a6a264760 P 9bf6780949014442ac25f7697760cfa8 [CANDIDATE]: Term 1 election: Requested vote from peers b4c32366ec2f49d9a4f57daf07e375b9 (127.23.167.194:43695), 6e3b64b877e746ed97cc9c057006200f (127.23.167.195:35027)
I20260928 08:53:29.559971 28704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4d09a3d9f9745a0be55687a6a264760" candidate_uuid: "9bf6780949014442ac25f7697760cfa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e3b64b877e746ed97cc9c057006200f"
I20260928 08:53:29.560015 28571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4d09a3d9f9745a0be55687a6a264760" candidate_uuid: "9bf6780949014442ac25f7697760cfa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4c32366ec2f49d9a4f57daf07e375b9"
I20260928 08:53:29.560045 28704 raft_consensus.cc:3037] T c4d09a3d9f9745a0be55687a6a264760 P 6e3b64b877e746ed97cc9c057006200f [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:29.560089 28571 raft_consensus.cc:3037] T c4d09a3d9f9745a0be55687a6a264760 P b4c32366ec2f49d9a4f57daf07e375b9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:29.560751 28571 raft_consensus.cc:2454] T c4d09a3d9f9745a0be55687a6a264760 P b4c32366ec2f49d9a4f57daf07e375b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bf6780949014442ac25f7697760cfa8 in term 1.
I20260928 08:53:29.560758 28704 raft_consensus.cc:2454] T c4d09a3d9f9745a0be55687a6a264760 P 6e3b64b877e746ed97cc9c057006200f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bf6780949014442ac25f7697760cfa8 in term 1.
I20260928 08:53:29.560954 28372 leader_election.cc:302] T c4d09a3d9f9745a0be55687a6a264760 P 9bf6780949014442ac25f7697760cfa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bf6780949014442ac25f7697760cfa8, b4c32366ec2f49d9a4f57daf07e375b9; no voters:
I20260928 08:53:29.561115 28772 raft_consensus.cc:2781] T c4d09a3d9f9745a0be55687a6a264760 P 9bf6780949014442ac25f7697760cfa8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:53:29.561268 28772 raft_consensus.cc:686] T c4d09a3d9f9745a0be55687a6a264760 P 9bf6780949014442ac25f7697760cfa8 [term 1 LEADER]: Becoming Leader. State: Replica: 9bf6780949014442ac25f7697760cfa8, State: Running, Role: LEADER
I20260928 08:53:29.561431 28772 consensus_queue.cc:237] T c4d09a3d9f9745a0be55687a6a264760 P 9bf6780949014442ac25f7697760cfa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c32366ec2f49d9a4f57daf07e375b9" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 43695 } } peers { permanent_uuid: "9bf6780949014442ac25f7697760cfa8" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 45749 } } peers { permanent_uuid: "6e3b64b877e746ed97cc9c057006200f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 35027 } }
I20260928 08:53:29.562269 28293 catalog_manager.cc:5495] T c4d09a3d9f9745a0be55687a6a264760 P 9bf6780949014442ac25f7697760cfa8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bf6780949014442ac25f7697760cfa8 (127.23.167.193). New cstate: current_term: 1 leader_uuid: "9bf6780949014442ac25f7697760cfa8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c32366ec2f49d9a4f57daf07e375b9" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 43695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bf6780949014442ac25f7697760cfa8" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e3b64b877e746ed97cc9c057006200f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 35027 } health_report { overall_health: UNKNOWN } } }
W20260928 08:53:29.588109 28751 tablet.cc:2369] T c4d09a3d9f9745a0be55687a6a264760 P 6e3b64b877e746ed97cc9c057006200f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:53:29.597481 28763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:29.595196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45476 (local address 127.23.167.193:45749)
0928 08:53:29.595250 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:29.595252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:29.595258 (+ 6us) server_negotiation.cc:407] Connection header received
0928 08:53:29.595295 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:29.595296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:29.595314 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:29.595375 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:53:29.595483 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.596008 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.596738 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:29.596928 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:29.596975 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:29.597126 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:53:29.597234 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:53:29.597296 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:29.597384 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 08:53:29.597402 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 08:53:29.599381 28704 raft_consensus.cc:1261] T c4d09a3d9f9745a0be55687a6a264760 P 6e3b64b877e746ed97cc9c057006200f [term 1 FOLLOWER]: Refusing update from remote peer 9bf6780949014442ac25f7697760cfa8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:53:29.599380 28571 raft_consensus.cc:1261] T c4d09a3d9f9745a0be55687a6a264760 P b4c32366ec2f49d9a4f57daf07e375b9 [term 1 FOLLOWER]: Refusing update from remote peer 9bf6780949014442ac25f7697760cfa8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:53:29.599730 28772 consensus_queue.cc:1035] T c4d09a3d9f9745a0be55687a6a264760 P 9bf6780949014442ac25f7697760cfa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4c32366ec2f49d9a4f57daf07e375b9" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:53:29.600100 28772 consensus_queue.cc:1035] T c4d09a3d9f9745a0be55687a6a264760 P 9bf6780949014442ac25f7697760cfa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e3b64b877e746ed97cc9c057006200f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 35027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:53:29.602504 28778 mvcc.cc:205] Tried to move back new op lower bound from 7334238656916295680 to 7334238656764436480. Current Snapshot: MvccSnapshot[applied={T|T < 7334238656916295680}]
I20260928 08:53:29.602939 28779 mvcc.cc:205] Tried to move back new op lower bound from 7334238656916295680 to 7334238656764436480. Current Snapshot: MvccSnapshot[applied={T|T < 7334238656916295680}]
I20260928 08:53:29.605458 28293 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:53:29.605574 28293 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:53:29.606444 28293 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 872ecf4a0a654ed6bfd50f8651eee89a: Sending DeleteTablet for 3 replicas of tablet c4d09a3d9f9745a0be55687a6a264760
I20260928 08:53:29.606808 28684 tablet_service.cc:1514] Processing DeleteTablet for tablet c4d09a3d9f9745a0be55687a6a264760 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:53:29 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:42892
I20260928 08:53:29.606819 28418 tablet_service.cc:1514] Processing DeleteTablet for tablet c4d09a3d9f9745a0be55687a6a264760 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:53:29 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:45464
I20260928 08:53:29.607210 28788 tablet_replica.cc:330] T c4d09a3d9f9745a0be55687a6a264760 P 6e3b64b877e746ed97cc9c057006200f: stopping tablet replica
I20260928 08:53:29.607183 28551 tablet_service.cc:1514] Processing DeleteTablet for tablet c4d09a3d9f9745a0be55687a6a264760 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:53:29 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:53200
I20260928 08:53:29.607326 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 28356
I20260928 08:53:29.607389 28788 raft_consensus.cc:2229] T c4d09a3d9f9745a0be55687a6a264760 P 6e3b64b877e746ed97cc9c057006200f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:53:29.607506 28788 raft_consensus.cc:2258] T c4d09a3d9f9745a0be55687a6a264760 P 6e3b64b877e746ed97cc9c057006200f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:53:29.608108 28788 ts_tablet_manager.cc:1905] T c4d09a3d9f9745a0be55687a6a264760 P 6e3b64b877e746ed97cc9c057006200f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:53:29.608605 28790 tablet_replica.cc:330] T c4d09a3d9f9745a0be55687a6a264760 P b4c32366ec2f49d9a4f57daf07e375b9: stopping tablet replica
I20260928 08:53:29.608741 28790 raft_consensus.cc:2229] T c4d09a3d9f9745a0be55687a6a264760 P b4c32366ec2f49d9a4f57daf07e375b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:53:29.608855 28790 raft_consensus.cc:2258] T c4d09a3d9f9745a0be55687a6a264760 P b4c32366ec2f49d9a4f57daf07e375b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:53:29.609825 28788 ts_tablet_manager.cc:1918] T c4d09a3d9f9745a0be55687a6a264760 P 6e3b64b877e746ed97cc9c057006200f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:53:29.609913 28788 log.cc:1196] T c4d09a3d9f9745a0be55687a6a264760 P 6e3b64b877e746ed97cc9c057006200f: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-2/wal/wals/c4d09a3d9f9745a0be55687a6a264760
I20260928 08:53:29.610082 28790 ts_tablet_manager.cc:1905] T c4d09a3d9f9745a0be55687a6a264760 P b4c32366ec2f49d9a4f57daf07e375b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:53:29.610198 28788 ts_tablet_manager.cc:1939] T c4d09a3d9f9745a0be55687a6a264760 P 6e3b64b877e746ed97cc9c057006200f: Deleting consensus metadata
I20260928 08:53:29.610707 28278 catalog_manager.cc:4841] TS 6e3b64b877e746ed97cc9c057006200f (127.23.167.195:35027): tablet c4d09a3d9f9745a0be55687a6a264760 (table test-table [id=cd896c7708154f10aa95eec5df5c482a]) successfully deleted
I20260928 08:53:29.611881 28790 ts_tablet_manager.cc:1918] T c4d09a3d9f9745a0be55687a6a264760 P b4c32366ec2f49d9a4f57daf07e375b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:53:29.611950 28790 log.cc:1196] T c4d09a3d9f9745a0be55687a6a264760 P b4c32366ec2f49d9a4f57daf07e375b9: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585589999956-24223-0/minicluster-data/ts-1/wal/wals/c4d09a3d9f9745a0be55687a6a264760
I20260928 08:53:29.612160 28790 ts_tablet_manager.cc:1939] T c4d09a3d9f9745a0be55687a6a264760 P b4c32366ec2f49d9a4f57daf07e375b9: Deleting consensus metadata
I20260928 08:53:29.612607 28279 catalog_manager.cc:4841] TS b4c32366ec2f49d9a4f57daf07e375b9 (127.23.167.194:43695): tablet c4d09a3d9f9745a0be55687a6a264760 (table test-table [id=cd896c7708154f10aa95eec5df5c482a]) successfully deleted
W20260928 08:53:29.612963 28280 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.167.193:45749 (error 108)
I20260928 08:53:29.613147 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 28489
I20260928 08:53:29.614102 28760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:29.613964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:45749 (local address 127.0.0.1:45488)
0928 08:53:29.614027 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:29.614072 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.193:45749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 08:53:29.614233 28280 catalog_manager.cc:4584] TS 9bf6780949014442ac25f7697760cfa8 (127.23.167.193:45749): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c4d09a3d9f9745a0be55687a6a264760: Network error: Client connection negotiation failed: client connection to 127.23.167.193:45749: connect: Connection refused (error 111)
I20260928 08:53:29.617853 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 28622
I20260928 08:53:29.622490 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 28263
2026-09-28T08:53:29Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1967 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T08:53:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:53:29Z Disabled control of system clock
I20260928 08:53:29.648332 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:43307
--webserver_interface=127.23.167.254
--webserver_port=0
--builtin_ntp_servers=127.23.167.212:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:43307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 08:53:29.728698 28795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:29.728986 28795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:29.729018 28795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:29.730662 28795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:53:29.730711 28795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:29.730727 28795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:29.730742 28795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:53:29.730759 28795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:53:29.732684 28795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:43307
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:43307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:29.732967 28795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:29.733319 28795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:29.739126 28801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:29.739137 28800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:29.739138 28803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:29.739653 28795 server_base.cc:1034] running on GCE node
I20260928 08:53:29.739913 28795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:29.740247 28795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:29.741397 28795 hybrid_clock.cc:648] HybridClock initialized: now 1790585609741374 us; error 35 us; skew 500 ppm
I20260928 08:53:29.742626 28795 webserver.cc:460] Webserver started at http://127.23.167.254:42083/ using document root <none> and password file <none>
I20260928 08:53:29.742842 28795 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:29.742879 28795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:29.742986 28795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:29.744076 28795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "08fd5469ddaa430a9c2463c2ae9b697f"
format_stamp: "Formatted at 2026-09-28 08:53:29 on dist-test-slave-pbjp"
server_key: "0f74251e71d8c7fb0400979bc7b00bfe"
server_key_iv: "8e2786ea055b3c51ccb59b7e8bc65193"
server_key_version: "encryptionkey@0"
I20260928 08:53:29.744361 28795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "08fd5469ddaa430a9c2463c2ae9b697f"
format_stamp: "Formatted at 2026-09-28 08:53:29 on dist-test-slave-pbjp"
server_key: "0f74251e71d8c7fb0400979bc7b00bfe"
server_key_iv: "8e2786ea055b3c51ccb59b7e8bc65193"
server_key_version: "encryptionkey@0"
I20260928 08:53:29.745648 28795 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 08:53:29.746630 28809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:29.746778 28795 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:53:29.746866 28795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "08fd5469ddaa430a9c2463c2ae9b697f"
format_stamp: "Formatted at 2026-09-28 08:53:29 on dist-test-slave-pbjp"
server_key: "0f74251e71d8c7fb0400979bc7b00bfe"
server_key_iv: "8e2786ea055b3c51ccb59b7e8bc65193"
server_key_version: "encryptionkey@0"
I20260928 08:53:29.746935 28795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:29.751787 28795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:29.752101 28795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:29.752209 28795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:29.756685 28795 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:43307
I20260928 08:53:29.756726 28861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:43307 every 8 connection(s)
I20260928 08:53:29.757087 28795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:53:29.757876 28862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:29.762287 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 28795
I20260928 08:53:29.762390 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:53:29.762562 24223 external_mini_cluster.cc:1428] Setting key 255e0f345bf2edd12e2abdb1ed9a21d4
I20260928 08:53:29.762728 28862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08fd5469ddaa430a9c2463c2ae9b697f: Bootstrap starting.
I20260928 08:53:29.763723 28862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08fd5469ddaa430a9c2463c2ae9b697f: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:29.764082 28862 log.cc:824] T 00000000000000000000000000000000 P 08fd5469ddaa430a9c2463c2ae9b697f: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:29.764832 28862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08fd5469ddaa430a9c2463c2ae9b697f: No bootstrap required, opened a new log
I20260928 08:53:29.766848 28862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08fd5469ddaa430a9c2463c2ae9b697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fd5469ddaa430a9c2463c2ae9b697f" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43307 } }
I20260928 08:53:29.767040 28862 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08fd5469ddaa430a9c2463c2ae9b697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:29.767073 28862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08fd5469ddaa430a9c2463c2ae9b697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08fd5469ddaa430a9c2463c2ae9b697f, State: Initialized, Role: FOLLOWER
I20260928 08:53:29.767196 28862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08fd5469ddaa430a9c2463c2ae9b697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fd5469ddaa430a9c2463c2ae9b697f" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43307 } }
I20260928 08:53:29.767279 28862 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08fd5469ddaa430a9c2463c2ae9b697f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:53:29.767311 28862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08fd5469ddaa430a9c2463c2ae9b697f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:53:29.767351 28862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08fd5469ddaa430a9c2463c2ae9b697f [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:29.768069 28862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08fd5469ddaa430a9c2463c2ae9b697f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fd5469ddaa430a9c2463c2ae9b697f" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43307 } }
I20260928 08:53:29.768170 28862 leader_election.cc:302] T 00000000000000000000000000000000 P 08fd5469ddaa430a9c2463c2ae9b697f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08fd5469ddaa430a9c2463c2ae9b697f; no voters:
I20260928 08:53:29.768354 28862 leader_election.cc:288] T 00000000000000000000000000000000 P 08fd5469ddaa430a9c2463c2ae9b697f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:53:29.768380 28867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08fd5469ddaa430a9c2463c2ae9b697f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:53:29.768491 28867 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08fd5469ddaa430a9c2463c2ae9b697f [term 1 LEADER]: Becoming Leader. State: Replica: 08fd5469ddaa430a9c2463c2ae9b697f, State: Running, Role: LEADER
I20260928 08:53:29.768577 28867 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08fd5469ddaa430a9c2463c2ae9b697f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fd5469ddaa430a9c2463c2ae9b697f" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43307 } }
I20260928 08:53:29.768627 28862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08fd5469ddaa430a9c2463c2ae9b697f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:53:29.768886 28867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08fd5469ddaa430a9c2463c2ae9b697f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08fd5469ddaa430a9c2463c2ae9b697f. Latest consensus state: current_term: 1 leader_uuid: "08fd5469ddaa430a9c2463c2ae9b697f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fd5469ddaa430a9c2463c2ae9b697f" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43307 } } }
I20260928 08:53:29.768959 28867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08fd5469ddaa430a9c2463c2ae9b697f [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:29.768898 28868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08fd5469ddaa430a9c2463c2ae9b697f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08fd5469ddaa430a9c2463c2ae9b697f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fd5469ddaa430a9c2463c2ae9b697f" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43307 } } }
I20260928 08:53:29.769011 28868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08fd5469ddaa430a9c2463c2ae9b697f [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:29.770223 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:29.763038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54938 (local address 127.23.167.254:43307)
0928 08:53:29.763247 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:29.763256 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:29.763290 (+ 34us) server_negotiation.cc:407] Connection header received
0928 08:53:29.763895 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:29.763905 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:29.763965 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:29.764119 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:29.769587 (+ 5468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:29.769591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:29.769592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:53:29.769597 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:29.769644 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:53:29.769659 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:29.769734 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:29.769963 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 08:53:29.770035 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
W20260928 08:53:29.771747 28883 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 08fd5469ddaa430a9c2463c2ae9b697f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:53:29.771798 28883 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:53:29.771839 28871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:53:29.772028 28871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:53:29.773615 28871 catalog_manager.cc:1347] Generated new cluster ID: ee72ec3ebfd14817b26c7894e77e5675
I20260928 08:53:29.773671 28871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:53:29.784945 28871 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:53:29.785142 28871 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:53:29.789012 28871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08fd5469ddaa430a9c2463c2ae9b697f: Generated new TSK 0
I20260928 08:53:29.789208 28871 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:53:29.794049 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:0
--local_ip_for_outbound_sockets=127.23.167.193
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:43307
--builtin_ntp_servers=127.23.167.212:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 08:53:29.875340 28886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:29.875540 28886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:29.875560 28886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:29.876711 28886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:29.876750 28886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:29.876775 28886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:53:29.878146 28886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:43307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:29.878382 28886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:29.878675 28886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:29.884413 28894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:29.884397 28891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:29.884632 28886 server_base.cc:1034] running on GCE node
W20260928 08:53:29.884393 28892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:29.884953 28886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:29.885288 28886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:29.886462 28886 hybrid_clock.cc:648] HybridClock initialized: now 1790585609886449 us; error 29 us; skew 500 ppm
I20260928 08:53:29.887825 28886 webserver.cc:460] Webserver started at http://127.23.167.193:35733/ using document root <none> and password file <none>
I20260928 08:53:29.888108 28886 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:29.888161 28886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:29.888281 28886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:29.889636 28886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-0/data/instance:
uuid: "c6fc180925c541e18891d120ba0f9f72"
format_stamp: "Formatted at 2026-09-28 08:53:29 on dist-test-slave-pbjp"
server_key: "79200165f7de95dea0d7c3186fc4d8ff"
server_key_iv: "3e0be3cf8964b59b05ff3424f23aea51"
server_key_version: "encryptionkey@0"
I20260928 08:53:29.890017 28886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance:
uuid: "c6fc180925c541e18891d120ba0f9f72"
format_stamp: "Formatted at 2026-09-28 08:53:29 on dist-test-slave-pbjp"
server_key: "79200165f7de95dea0d7c3186fc4d8ff"
server_key_iv: "3e0be3cf8964b59b05ff3424f23aea51"
server_key_version: "encryptionkey@0"
I20260928 08:53:29.891544 28886 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:53:29.892349 28900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:29.892500 28886 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:53:29.892582 28886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "c6fc180925c541e18891d120ba0f9f72"
format_stamp: "Formatted at 2026-09-28 08:53:29 on dist-test-slave-pbjp"
server_key: "79200165f7de95dea0d7c3186fc4d8ff"
server_key_iv: "3e0be3cf8964b59b05ff3424f23aea51"
server_key_version: "encryptionkey@0"
I20260928 08:53:29.892652 28886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:29.897677 28886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:29.898037 28886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:29.898208 28886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:29.898455 28886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:53:29.898819 28886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:29.898864 28886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:29.898902 28886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:29.898926 28886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:29.905673 28886 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:39481
I20260928 08:53:29.905706 29013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:39481 every 8 connection(s)
I20260928 08:53:29.906144 28886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
I20260928 08:53:29.908414 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 28886
I20260928 08:53:29.908560 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance
I20260928 08:53:29.908752 24223 external_mini_cluster.cc:1428] Setting key 530a2b4fddf4bff48afde93245eef2d5
I20260928 08:53:29.909412 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:29.907022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:50289 (local address 127.23.167.254:43307)
0928 08:53:29.907077 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:29.907078 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:29.907314 (+ 236us) server_negotiation.cc:407] Connection header received
0928 08:53:29.907782 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:29.907783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:29.907790 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:29.907838 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:29.909188 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:29.909189 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:29.909189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:53:29.909194 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:29.909210 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:53:29.909214 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:29.909259 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:29.909329 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 08:53:29.909344 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 08:53:29.909610 29016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:29.906996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:43307 (local address 127.23.167.193:50289)
0928 08:53:29.907191 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:29.907224 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:29.907690 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:29.907911 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:29.907915 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:29.907923 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:29.908981 (+ 1058us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:53:29.908981 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:29.909014 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:53:29.909050 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:29.909263 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:29.909269 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:29.909272 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:53:29.909343 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 08:53:29.909405 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260928 08:53:29.909950 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:0
--local_ip_for_outbound_sockets=127.23.167.194
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:43307
--builtin_ntp_servers=127.23.167.212:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 08:53:29.910029 29014 heartbeater.cc:347] Connected to a master server at 127.23.167.254:43307
I20260928 08:53:29.910125 29014 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:29.910260 29014 heartbeater.cc:511] Master 127.23.167.254:43307 requested a full tablet report, sending...
I20260928 08:53:29.910758 28826 ts_manager.cc:194] Registered new tserver with Master: c6fc180925c541e18891d120ba0f9f72 (127.23.167.193:39481)
W20260928 08:53:29.988934 29017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:29.989156 29017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:29.989176 29017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:29.990326 29017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:29.990372 29017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:29.990398 29017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:53:29.991703 29017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:43307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:29.991921 29017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:29.992188 29017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:29.996888 29022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:29.996989 29017 server_base.cc:1034] running on GCE node
W20260928 08:53:29.996896 29025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:29.996905 29023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:29.997357 29017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:29.997596 29017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:29.998764 29017 hybrid_clock.cc:648] HybridClock initialized: now 1790585609998739 us; error 42 us; skew 500 ppm
I20260928 08:53:29.999967 29017 webserver.cc:460] Webserver started at http://127.23.167.194:44197/ using document root <none> and password file <none>
I20260928 08:53:30.000185 29017 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:30.000228 29017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:30.000352 29017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:30.001513 29017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-1/data/instance:
uuid: "3040bbe94988429cbc926e369d1f116a"
format_stamp: "Formatted at 2026-09-28 08:53:29 on dist-test-slave-pbjp"
server_key: "650bf9d5dd3402a29bff72c9a4918fa1"
server_key_iv: "f95445fdca9b6c3d3c20f867c22fe965"
server_key_version: "encryptionkey@0"
I20260928 08:53:30.001848 29017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance:
uuid: "3040bbe94988429cbc926e369d1f116a"
format_stamp: "Formatted at 2026-09-28 08:53:29 on dist-test-slave-pbjp"
server_key: "650bf9d5dd3402a29bff72c9a4918fa1"
server_key_iv: "f95445fdca9b6c3d3c20f867c22fe965"
server_key_version: "encryptionkey@0"
I20260928 08:53:30.003052 29017 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 08:53:30.003747 29031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:30.003906 29017 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:53:30.003994 29017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "3040bbe94988429cbc926e369d1f116a"
format_stamp: "Formatted at 2026-09-28 08:53:29 on dist-test-slave-pbjp"
server_key: "650bf9d5dd3402a29bff72c9a4918fa1"
server_key_iv: "f95445fdca9b6c3d3c20f867c22fe965"
server_key_version: "encryptionkey@0"
I20260928 08:53:30.004057 29017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:30.007339 29017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:30.007588 29017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:30.007714 29017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:30.007900 29017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:53:30.008191 29017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:30.008224 29017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:30.008251 29017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:30.008275 29017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:30.013948 29017 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:35773
I20260928 08:53:30.013980 29144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:35773 every 8 connection(s)
I20260928 08:53:30.014323 29017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
I20260928 08:53:30.017167 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:30.015054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:37121 (local address 127.23.167.254:43307)
0928 08:53:30.015124 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:30.015126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:30.015306 (+ 180us) server_negotiation.cc:407] Connection header received
0928 08:53:30.015759 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:30.015760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:30.015765 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:30.015812 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:30.016897 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:30.016898 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:30.016898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:53:30.016900 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:30.016910 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:53:30.016913 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:30.016960 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:30.017023 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 08:53:30.017033 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 08:53:30.017300 29147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:30.015029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:43307 (local address 127.23.167.194:37121)
0928 08:53:30.015192 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:30.015225 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:30.015681 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:30.015877 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:30.015881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:30.015889 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:30.016773 (+ 884us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:53:30.016773 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:30.016804 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:53:30.016833 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:30.016964 (+ 131us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:30.016966 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:30.016969 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:53:30.017027 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 08:53:30.017119 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260928 08:53:30.017742 29145 heartbeater.cc:347] Connected to a master server at 127.23.167.254:43307
I20260928 08:53:30.017828 29145 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:30.017969 29145 heartbeater.cc:511] Master 127.23.167.254:43307 requested a full tablet report, sending...
I20260928 08:53:30.018296 28826 ts_manager.cc:194] Registered new tserver with Master: 3040bbe94988429cbc926e369d1f116a (127.23.167.194:35773)
I20260928 08:53:30.024564 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 29017
I20260928 08:53:30.024667 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance
I20260928 08:53:30.024829 24223 external_mini_cluster.cc:1428] Setting key 4f21d3fff71e2888b1d558e38ebba58b
I20260928 08:53:30.025818 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:0
--local_ip_for_outbound_sockets=127.23.167.195
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:43307
--builtin_ntp_servers=127.23.167.212:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 08:53:30.105986 29148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:30.106197 29148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:30.106218 29148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:30.107440 29148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:30.107493 29148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:30.107527 29148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:53:30.108912 29148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:43307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:30.109176 29148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:30.109476 29148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:30.114181 29154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:30.114326 29148 server_base.cc:1034] running on GCE node
W20260928 08:53:30.114198 29153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:30.114181 29156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:30.114657 29148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:30.114894 29148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:30.116039 29148 hybrid_clock.cc:648] HybridClock initialized: now 1790585610116026 us; error 27 us; skew 500 ppm
I20260928 08:53:30.117316 29148 webserver.cc:460] Webserver started at http://127.23.167.195:34257/ using document root <none> and password file <none>
I20260928 08:53:30.117563 29148 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:30.117612 29148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:30.117729 29148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:30.118814 29148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-2/data/instance:
uuid: "d2d22fd6ae0146a0a07ae3706506481a"
format_stamp: "Formatted at 2026-09-28 08:53:30 on dist-test-slave-pbjp"
server_key: "ea3e65e79ef768168e3bfe7377b4f1e2"
server_key_iv: "fc068d452ef8ed46a607a8857322be70"
server_key_version: "encryptionkey@0"
I20260928 08:53:30.119155 29148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance:
uuid: "d2d22fd6ae0146a0a07ae3706506481a"
format_stamp: "Formatted at 2026-09-28 08:53:30 on dist-test-slave-pbjp"
server_key: "ea3e65e79ef768168e3bfe7377b4f1e2"
server_key_iv: "fc068d452ef8ed46a607a8857322be70"
server_key_version: "encryptionkey@0"
I20260928 08:53:30.120522 29148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:53:30.121311 29162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:30.121469 29148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:53:30.121567 29148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "d2d22fd6ae0146a0a07ae3706506481a"
format_stamp: "Formatted at 2026-09-28 08:53:30 on dist-test-slave-pbjp"
server_key: "ea3e65e79ef768168e3bfe7377b4f1e2"
server_key_iv: "fc068d452ef8ed46a607a8857322be70"
server_key_version: "encryptionkey@0"
I20260928 08:53:30.121637 29148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:30.124884 29148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:30.125177 29148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:30.125280 29148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:30.125486 29148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:53:30.125795 29148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:30.125829 29148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:30.125849 29148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:30.125871 29148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:30.131314 29148 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:45251
I20260928 08:53:30.131386 29275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:45251 every 8 connection(s)
I20260928 08:53:30.131685 29148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
I20260928 08:53:30.134913 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:30.132475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:37527 (local address 127.23.167.254:43307)
0928 08:53:30.132574 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:30.132577 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:30.132774 (+ 197us) server_negotiation.cc:407] Connection header received
0928 08:53:30.133375 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:30.133376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:30.133384 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:30.133444 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:30.134641 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:30.134642 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:30.134643 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:53:30.134648 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:30.134663 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:53:30.134667 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:30.134714 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:30.134819 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 08:53:30.134836 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 08:53:30.135126 29278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:30.132454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:43307 (local address 127.23.167.195:37527)
0928 08:53:30.132650 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:30.132685 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:30.133267 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:30.133525 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:30.133529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:30.133538 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:30.134460 (+ 922us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:53:30.134461 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:30.134494 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:53:30.134528 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:30.134741 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:30.134747 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:30.134750 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:53:30.134817 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 08:53:30.134908 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260928 08:53:30.135584 29276 heartbeater.cc:347] Connected to a master server at 127.23.167.254:43307
I20260928 08:53:30.135667 29276 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:30.135788 29276 heartbeater.cc:511] Master 127.23.167.254:43307 requested a full tablet report, sending...
I20260928 08:53:30.136121 28826 ts_manager.cc:194] Registered new tserver with Master: d2d22fd6ae0146a0a07ae3706506481a (127.23.167.195:45251)
I20260928 08:53:30.140302 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 29148
I20260928 08:53:30.140394 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance
I20260928 08:53:30.140555 24223 external_mini_cluster.cc:1428] Setting key c0144fcdb4dd423ca411d4595d9edbc8
I20260928 08:53:30.141425 24223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:53:30.143723 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:30.142730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54946 (local address 127.23.167.254:43307)
0928 08:53:30.142797 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:30.142798 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:30.142869 (+ 71us) server_negotiation.cc:407] Connection header received
0928 08:53:30.142936 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:30.142937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:30.142942 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:30.142978 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:30.143543 (+ 565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:30.143544 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:30.143544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:53:30.143548 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:30.143564 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:53:30.143568 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:30.143626 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:30.143660 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 08:53:30.143671 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 08:53:30.145358 28826 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:53:30.147119 28826 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:53:30.151054 29285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:30.149471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:35773 (local address 127.0.0.1:55234)
0928 08:53:30.149667 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:30.149712 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:30.149769 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:30.150101 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:30.150107 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:30.150114 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:30.150622 (+ 508us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:53:30.150623 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:30.150641 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:53:30.150689 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:30.150897 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:30.150903 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:30.150906 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:53:30.150967 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 08:53:30.150981 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":52,"threads_started":1}
I20260928 08:53:30.151180 29284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:30.149478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:39481 (local address 127.0.0.1:50608)
0928 08:53:30.149780 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:30.149790 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:30.149835 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:30.150501 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:30.150501 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:30.150505 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:30.150857 (+ 352us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:53:30.150858 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:30.150862 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:53:30.150872 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:30.151089 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:30.151090 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:30.151090 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 08:53:30.151117 (+ 27us) client_negotiation.cc:240] Negotiation successful
0928 08:53:30.151130 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":113,"threads_started":1}
I20260928 08:53:30.151324 29286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:30.149603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55234 (local address 127.23.167.194:35773)
0928 08:53:30.149859 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:30.149864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:30.149873 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:53:30.149897 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:30.149899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:30.149907 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:30.149971 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:30.150754 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:30.150763 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:30.150763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:53:30.150767 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:30.150805 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:53:30.150831 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:30.150900 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:30.151227 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 08:53:30.151244 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":85,"threads_started":1}
I20260928 08:53:30.151441 29288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:30.149651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50608 (local address 127.23.167.193:39481)
0928 08:53:30.150357 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:30.150360 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:30.150369 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:53:30.150394 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:30.150396 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:30.150426 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:30.150496 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:30.150943 (+ 447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:30.150949 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:30.150950 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:53:30.150954 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:30.150989 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:53:30.151021 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:30.151328 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:30.151351 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 08:53:30.151365 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":90,"threads_started":1}
I20260928 08:53:30.151547 29287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:30.149653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:45251 (local address 127.0.0.1:54662)
0928 08:53:30.150042 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:30.150051 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:30.150099 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:30.150585 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:30.150586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:30.150587 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:30.151204 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:53:30.151204 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:30.151211 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:53:30.151223 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:30.151428 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:30.151429 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:30.151430 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 08:53:30.151466 (+ 36us) client_negotiation.cc:240] Negotiation successful
0928 08:53:30.151478 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":64,"threads_started":1}
I20260928 08:53:30.151808 29289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:30.149873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54662 (local address 127.23.167.195:45251)
0928 08:53:30.150036 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:30.150041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:30.150149 (+ 108us) server_negotiation.cc:407] Connection header received
0928 08:53:30.150175 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:30.150179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:30.150189 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:30.150255 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:30.151296 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:30.151301 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:30.151301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:53:30.151305 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:30.151334 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:53:30.151354 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:30.151405 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:30.151698 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 08:53:30.151714 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260928 08:53:30.152045 29079 tablet_service.cc:1467] Processing CreateTablet for tablet 911fdbb6a14748a2ab1424c496d43a26 (DEFAULT_TABLE table=test-table [id=edea594686aa474497312ff822d50e3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:53:30.152431 29079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 911fdbb6a14748a2ab1424c496d43a26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:30.152446 29210 tablet_service.cc:1467] Processing CreateTablet for tablet 911fdbb6a14748a2ab1424c496d43a26 (DEFAULT_TABLE table=test-table [id=edea594686aa474497312ff822d50e3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:53:30.152776 29210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 911fdbb6a14748a2ab1424c496d43a26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:30.155800 29290 tablet_bootstrap.cc:492] T 911fdbb6a14748a2ab1424c496d43a26 P 3040bbe94988429cbc926e369d1f116a: Bootstrap starting.
I20260928 08:53:30.156176 28948 tablet_service.cc:1467] Processing CreateTablet for tablet 911fdbb6a14748a2ab1424c496d43a26 (DEFAULT_TABLE table=test-table [id=edea594686aa474497312ff822d50e3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:53:30.156428 29290 tablet_bootstrap.cc:654] T 911fdbb6a14748a2ab1424c496d43a26 P 3040bbe94988429cbc926e369d1f116a: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:30.156464 28948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 911fdbb6a14748a2ab1424c496d43a26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:30.156721 29290 log.cc:824] T 911fdbb6a14748a2ab1424c496d43a26 P 3040bbe94988429cbc926e369d1f116a: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:30.157037 29291 tablet_bootstrap.cc:492] T 911fdbb6a14748a2ab1424c496d43a26 P d2d22fd6ae0146a0a07ae3706506481a: Bootstrap starting.
I20260928 08:53:30.157593 29290 tablet_bootstrap.cc:492] T 911fdbb6a14748a2ab1424c496d43a26 P 3040bbe94988429cbc926e369d1f116a: No bootstrap required, opened a new log
I20260928 08:53:30.157667 29290 ts_tablet_manager.cc:1397] T 911fdbb6a14748a2ab1424c496d43a26 P 3040bbe94988429cbc926e369d1f116a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:53:30.157774 29291 tablet_bootstrap.cc:654] T 911fdbb6a14748a2ab1424c496d43a26 P d2d22fd6ae0146a0a07ae3706506481a: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:30.158033 29291 log.cc:824] T 911fdbb6a14748a2ab1424c496d43a26 P d2d22fd6ae0146a0a07ae3706506481a: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:30.158629 29291 tablet_bootstrap.cc:492] T 911fdbb6a14748a2ab1424c496d43a26 P d2d22fd6ae0146a0a07ae3706506481a: No bootstrap required, opened a new log
I20260928 08:53:30.158686 29291 ts_tablet_manager.cc:1397] T 911fdbb6a14748a2ab1424c496d43a26 P d2d22fd6ae0146a0a07ae3706506481a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 08:53:30.159711 29294 tablet_bootstrap.cc:492] T 911fdbb6a14748a2ab1424c496d43a26 P c6fc180925c541e18891d120ba0f9f72: Bootstrap starting.
I20260928 08:53:30.160135 29290 raft_consensus.cc:348] T 911fdbb6a14748a2ab1424c496d43a26 P 3040bbe94988429cbc926e369d1f116a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3040bbe94988429cbc926e369d1f116a" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 35773 } } peers { permanent_uuid: "c6fc180925c541e18891d120ba0f9f72" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 39481 } } peers { permanent_uuid: "d2d22fd6ae0146a0a07ae3706506481a" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45251 } }
I20260928 08:53:30.160269 29290 raft_consensus.cc:374] T 911fdbb6a14748a2ab1424c496d43a26 P 3040bbe94988429cbc926e369d1f116a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:30.160290 29290 raft_consensus.cc:729] T 911fdbb6a14748a2ab1424c496d43a26 P 3040bbe94988429cbc926e369d1f116a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3040bbe94988429cbc926e369d1f116a, State: Initialized, Role: FOLLOWER
I20260928 08:53:30.160403 29290 consensus_queue.cc:260] T 911fdbb6a14748a2ab1424c496d43a26 P 3040bbe94988429cbc926e369d1f116a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3040bbe94988429cbc926e369d1f116a" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 35773 } } peers { permanent_uuid: "c6fc180925c541e18891d120ba0f9f72" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 39481 } } peers { permanent_uuid: "d2d22fd6ae0146a0a07ae3706506481a" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45251 } }
I20260928 08:53:30.160652 29290 ts_tablet_manager.cc:1428] T 911fdbb6a14748a2ab1424c496d43a26 P 3040bbe94988429cbc926e369d1f116a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:53:30.160668 29145 heartbeater.cc:503] Master 127.23.167.254:43307 was elected leader, sending a full tablet report...
I20260928 08:53:30.160733 29294 tablet_bootstrap.cc:654] T 911fdbb6a14748a2ab1424c496d43a26 P c6fc180925c541e18891d120ba0f9f72: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:30.161113 29294 log.cc:824] T 911fdbb6a14748a2ab1424c496d43a26 P c6fc180925c541e18891d120ba0f9f72: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:30.161038 29291 raft_consensus.cc:348] T 911fdbb6a14748a2ab1424c496d43a26 P d2d22fd6ae0146a0a07ae3706506481a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3040bbe94988429cbc926e369d1f116a" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 35773 } } peers { permanent_uuid: "c6fc180925c541e18891d120ba0f9f72" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 39481 } } peers { permanent_uuid: "d2d22fd6ae0146a0a07ae3706506481a" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45251 } }
I20260928 08:53:30.161219 29291 raft_consensus.cc:374] T 911fdbb6a14748a2ab1424c496d43a26 P d2d22fd6ae0146a0a07ae3706506481a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:30.161257 29291 raft_consensus.cc:729] T 911fdbb6a14748a2ab1424c496d43a26 P d2d22fd6ae0146a0a07ae3706506481a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d22fd6ae0146a0a07ae3706506481a, State: Initialized, Role: FOLLOWER
I20260928 08:53:30.161377 29291 consensus_queue.cc:260] T 911fdbb6a14748a2ab1424c496d43a26 P d2d22fd6ae0146a0a07ae3706506481a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3040bbe94988429cbc926e369d1f116a" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 35773 } } peers { permanent_uuid: "c6fc180925c541e18891d120ba0f9f72" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 39481 } } peers { permanent_uuid: "d2d22fd6ae0146a0a07ae3706506481a" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45251 } }
I20260928 08:53:30.161603 29291 ts_tablet_manager.cc:1428] T 911fdbb6a14748a2ab1424c496d43a26 P d2d22fd6ae0146a0a07ae3706506481a: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 08:53:30.161641 29276 heartbeater.cc:503] Master 127.23.167.254:43307 was elected leader, sending a full tablet report...
I20260928 08:53:30.162218 29294 tablet_bootstrap.cc:492] T 911fdbb6a14748a2ab1424c496d43a26 P c6fc180925c541e18891d120ba0f9f72: No bootstrap required, opened a new log
I20260928 08:53:30.162288 29294 ts_tablet_manager.cc:1397] T 911fdbb6a14748a2ab1424c496d43a26 P c6fc180925c541e18891d120ba0f9f72: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260928 08:53:30.164130 29294 raft_consensus.cc:348] T 911fdbb6a14748a2ab1424c496d43a26 P c6fc180925c541e18891d120ba0f9f72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3040bbe94988429cbc926e369d1f116a" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 35773 } } peers { permanent_uuid: "c6fc180925c541e18891d120ba0f9f72" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 39481 } } peers { permanent_uuid: "d2d22fd6ae0146a0a07ae3706506481a" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45251 } }
I20260928 08:53:30.164251 29294 raft_consensus.cc:374] T 911fdbb6a14748a2ab1424c496d43a26 P c6fc180925c541e18891d120ba0f9f72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:30.164270 29294 raft_consensus.cc:729] T 911fdbb6a14748a2ab1424c496d43a26 P c6fc180925c541e18891d120ba0f9f72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6fc180925c541e18891d120ba0f9f72, State: Initialized, Role: FOLLOWER
I20260928 08:53:30.164346 29294 consensus_queue.cc:260] T 911fdbb6a14748a2ab1424c496d43a26 P c6fc180925c541e18891d120ba0f9f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3040bbe94988429cbc926e369d1f116a" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 35773 } } peers { permanent_uuid: "c6fc180925c541e18891d120ba0f9f72" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 39481 } } peers { permanent_uuid: "d2d22fd6ae0146a0a07ae3706506481a" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45251 } }
I20260928 08:53:30.164526 29294 ts_tablet_manager.cc:1428] T 911fdbb6a14748a2ab1424c496d43a26 P c6fc180925c541e18891d120ba0f9f72: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 08:53:30.164595 29014 heartbeater.cc:503] Master 127.23.167.254:43307 was elected leader, sending a full tablet report...
I20260928 08:53:30.175706 29295 raft_consensus.cc:482] T 911fdbb6a14748a2ab1424c496d43a26 P 3040bbe94988429cbc926e369d1f116a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:53:30.175830 29295 raft_consensus.cc:504] T 911fdbb6a14748a2ab1424c496d43a26 P 3040bbe94988429cbc926e369d1f116a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3040bbe94988429cbc926e369d1f116a" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 35773 } } peers { permanent_uuid: "c6fc180925c541e18891d120ba0f9f72" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 39481 } } peers { permanent_uuid: "d2d22fd6ae0146a0a07ae3706506481a" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45251 } }
I20260928 08:53:30.176146 29295 leader_election.cc:288] T 911fdbb6a14748a2ab1424c496d43a26 P 3040bbe94988429cbc926e369d1f116a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6fc180925c541e18891d120ba0f9f72 (127.23.167.193:39481), d2d22fd6ae0146a0a07ae3706506481a (127.23.167.195:45251)
I20260928 08:53:30.177325 29147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:30.176346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:39481 (local address 127.23.167.194:54381)
0928 08:53:30.176414 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:30.176426 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:30.176478 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:30.176639 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:30.176640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:30.176643 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:30.177044 (+ 401us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:53:30.177045 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:30.177054 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:53:30.177085 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:30.177215 (+ 130us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:30.177216 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:30.177216 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 08:53:30.177261 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 08:53:30.177275 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 08:53:30.177356 29288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:30.176382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:54381 (local address 127.23.167.193:39481)
0928 08:53:30.176429 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:30.176432 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:30.176522 (+ 90us) server_negotiation.cc:407] Connection header received
0928 08:53:30.176555 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:30.176556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:30.176564 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:30.176627 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:30.177155 (+ 528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:30.177156 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:30.177156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:53:30.177160 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:30.177175 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:53:30.177179 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:30.177215 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:30.177267 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 08:53:30.177282 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 08:53:30.177727 28968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "911fdbb6a14748a2ab1424c496d43a26" candidate_uuid: "3040bbe94988429cbc926e369d1f116a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6fc180925c541e18891d120ba0f9f72" is_pre_election: true
I20260928 08:53:30.177816 29289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:30.176535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:33057 (local address 127.23.167.195:45251)
0928 08:53:30.176585 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:30.176587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:30.176670 (+ 83us) server_negotiation.cc:407] Connection header received
0928 08:53:30.176754 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:30.176755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:30.176761 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:30.177050 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:30.177629 (+ 579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:30.177630 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:30.177630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:53:30.177639 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:30.177654 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:53:30.177658 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:30.177695 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:30.177759 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 08:53:30.177769 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 08:53:30.177846 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:30.176364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:45251 (local address 127.23.167.194:33057)
0928 08:53:30.176622 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:30.176630 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:30.176686 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:30.177090 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:30.177090 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:30.177093 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:30.177539 (+ 446us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:53:30.177540 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:30.177546 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:53:30.177558 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:30.177705 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:30.177706 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:30.177706 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 08:53:30.177760 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 08:53:30.177772 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":108,"threads_started":1}
I20260928 08:53:30.177906 28968 raft_consensus.cc:2454] T 911fdbb6a14748a2ab1424c496d43a26 P c6fc180925c541e18891d120ba0f9f72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3040bbe94988429cbc926e369d1f116a in term 0.
I20260928 08:53:30.178133 29032 leader_election.cc:302] T 911fdbb6a14748a2ab1424c496d43a26 P 3040bbe94988429cbc926e369d1f116a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3040bbe94988429cbc926e369d1f116a, c6fc180925c541e18891d120ba0f9f72; no voters:
I20260928 08:53:30.178143 29230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "911fdbb6a14748a2ab1424c496d43a26" candidate_uuid: "3040bbe94988429cbc926e369d1f116a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d22fd6ae0146a0a07ae3706506481a" is_pre_election: true
I20260928 08:53:30.178262 29230 raft_consensus.cc:2454] T 911fdbb6a14748a2ab1424c496d43a26 P d2d22fd6ae0146a0a07ae3706506481a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3040bbe94988429cbc926e369d1f116a in term 0.
I20260928 08:53:30.178288 29295 raft_consensus.cc:2781] T 911fdbb6a14748a2ab1424c496d43a26 P 3040bbe94988429cbc926e369d1f116a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:53:30.178339 29295 raft_consensus.cc:482] T 911fdbb6a14748a2ab1424c496d43a26 P 3040bbe94988429cbc926e369d1f116a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:53:30.178366 29295 raft_consensus.cc:3037] T 911fdbb6a14748a2ab1424c496d43a26 P 3040bbe94988429cbc926e369d1f116a [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:30.179080 29295 raft_consensus.cc:504] T 911fdbb6a14748a2ab1424c496d43a26 P 3040bbe94988429cbc926e369d1f116a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3040bbe94988429cbc926e369d1f116a" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 35773 } } peers { permanent_uuid: "c6fc180925c541e18891d120ba0f9f72" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 39481 } } peers { permanent_uuid: "d2d22fd6ae0146a0a07ae3706506481a" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45251 } }
I20260928 08:53:30.179193 29295 leader_election.cc:288] T 911fdbb6a14748a2ab1424c496d43a26 P 3040bbe94988429cbc926e369d1f116a [CANDIDATE]: Term 1 election: Requested vote from peers c6fc180925c541e18891d120ba0f9f72 (127.23.167.193:39481), d2d22fd6ae0146a0a07ae3706506481a (127.23.167.195:45251)
I20260928 08:53:30.179333 28968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "911fdbb6a14748a2ab1424c496d43a26" candidate_uuid: "3040bbe94988429cbc926e369d1f116a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6fc180925c541e18891d120ba0f9f72"
I20260928 08:53:30.179358 29230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "911fdbb6a14748a2ab1424c496d43a26" candidate_uuid: "3040bbe94988429cbc926e369d1f116a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d22fd6ae0146a0a07ae3706506481a"
I20260928 08:53:30.179409 28968 raft_consensus.cc:3037] T 911fdbb6a14748a2ab1424c496d43a26 P c6fc180925c541e18891d120ba0f9f72 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:30.179415 29230 raft_consensus.cc:3037] T 911fdbb6a14748a2ab1424c496d43a26 P d2d22fd6ae0146a0a07ae3706506481a [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:30.180251 28968 raft_consensus.cc:2454] T 911fdbb6a14748a2ab1424c496d43a26 P c6fc180925c541e18891d120ba0f9f72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3040bbe94988429cbc926e369d1f116a in term 1.
I20260928 08:53:30.180261 29230 raft_consensus.cc:2454] T 911fdbb6a14748a2ab1424c496d43a26 P d2d22fd6ae0146a0a07ae3706506481a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3040bbe94988429cbc926e369d1f116a in term 1.
I20260928 08:53:30.180413 29032 leader_election.cc:302] T 911fdbb6a14748a2ab1424c496d43a26 P 3040bbe94988429cbc926e369d1f116a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3040bbe94988429cbc926e369d1f116a, c6fc180925c541e18891d120ba0f9f72; no voters:
I20260928 08:53:30.180514 29295 raft_consensus.cc:2781] T 911fdbb6a14748a2ab1424c496d43a26 P 3040bbe94988429cbc926e369d1f116a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:53:30.180639 29295 raft_consensus.cc:686] T 911fdbb6a14748a2ab1424c496d43a26 P 3040bbe94988429cbc926e369d1f116a [term 1 LEADER]: Becoming Leader. State: Replica: 3040bbe94988429cbc926e369d1f116a, State: Running, Role: LEADER
I20260928 08:53:30.180712 29295 consensus_queue.cc:237] T 911fdbb6a14748a2ab1424c496d43a26 P 3040bbe94988429cbc926e369d1f116a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3040bbe94988429cbc926e369d1f116a" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 35773 } } peers { permanent_uuid: "c6fc180925c541e18891d120ba0f9f72" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 39481 } } peers { permanent_uuid: "d2d22fd6ae0146a0a07ae3706506481a" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45251 } }
I20260928 08:53:30.181545 28826 catalog_manager.cc:5495] T 911fdbb6a14748a2ab1424c496d43a26 P 3040bbe94988429cbc926e369d1f116a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3040bbe94988429cbc926e369d1f116a (127.23.167.194). New cstate: current_term: 1 leader_uuid: "3040bbe94988429cbc926e369d1f116a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3040bbe94988429cbc926e369d1f116a" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 35773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6fc180925c541e18891d120ba0f9f72" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2d22fd6ae0146a0a07ae3706506481a" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45251 } health_report { overall_health: UNKNOWN } } }
I20260928 08:53:30.186828 29286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:30.186107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55248 (local address 127.23.167.194:35773)
0928 08:53:30.186146 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:30.186147 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:30.186154 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:53:30.186178 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:30.186179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:30.186189 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:30.186230 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:30.186682 (+ 452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:30.186682 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:30.186682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:53:30.186685 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:30.186694 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:53:30.186696 (+ 2us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:30.186730 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:30.186760 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 08:53:30.186770 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 08:53:30.188441 28968 raft_consensus.cc:1261] T 911fdbb6a14748a2ab1424c496d43a26 P c6fc180925c541e18891d120ba0f9f72 [term 1 FOLLOWER]: Refusing update from remote peer 3040bbe94988429cbc926e369d1f116a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:53:30.188474 29230 raft_consensus.cc:1261] T 911fdbb6a14748a2ab1424c496d43a26 P d2d22fd6ae0146a0a07ae3706506481a [term 1 FOLLOWER]: Refusing update from remote peer 3040bbe94988429cbc926e369d1f116a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:53:30.188731 29295 consensus_queue.cc:1035] T 911fdbb6a14748a2ab1424c496d43a26 P 3040bbe94988429cbc926e369d1f116a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6fc180925c541e18891d120ba0f9f72" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:53:30.188879 29300 consensus_queue.cc:1035] T 911fdbb6a14748a2ab1424c496d43a26 P 3040bbe94988429cbc926e369d1f116a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d22fd6ae0146a0a07ae3706506481a" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 45251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:53:30.191741 29305 mvcc.cc:205] Tried to move back new op lower bound from 7334238659329351680 to 7334238659300880384. Current Snapshot: MvccSnapshot[applied={T|T < 7334238659329351680}]
I20260928 08:53:30.193624 28825 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:53:30.193706 28825 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:53:30.194407 28825 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08fd5469ddaa430a9c2463c2ae9b697f: Sending DeleteTablet for 3 replicas of tablet 911fdbb6a14748a2ab1424c496d43a26
I20260928 08:53:30.194727 29210 tablet_service.cc:1514] Processing DeleteTablet for tablet 911fdbb6a14748a2ab1424c496d43a26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:53:30 UTC) from {username='slave'} at 127.0.0.1:54662
I20260928 08:53:30.194728 28948 tablet_service.cc:1514] Processing DeleteTablet for tablet 911fdbb6a14748a2ab1424c496d43a26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:53:30 UTC) from {username='slave'} at 127.0.0.1:50608
I20260928 08:53:30.194936 29079 tablet_service.cc:1514] Processing DeleteTablet for tablet 911fdbb6a14748a2ab1424c496d43a26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:53:30 UTC) from {username='slave'} at 127.0.0.1:55234
I20260928 08:53:30.194957 29314 tablet_replica.cc:330] T 911fdbb6a14748a2ab1424c496d43a26 P c6fc180925c541e18891d120ba0f9f72: stopping tablet replica
I20260928 08:53:30.195055 29314 raft_consensus.cc:2229] T 911fdbb6a14748a2ab1424c496d43a26 P c6fc180925c541e18891d120ba0f9f72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:53:30.195158 29315 tablet_replica.cc:330] T 911fdbb6a14748a2ab1424c496d43a26 P 3040bbe94988429cbc926e369d1f116a: stopping tablet replica
I20260928 08:53:30.195170 29314 raft_consensus.cc:2258] T 911fdbb6a14748a2ab1424c496d43a26 P c6fc180925c541e18891d120ba0f9f72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:53:30.195259 29315 raft_consensus.cc:2229] T 911fdbb6a14748a2ab1424c496d43a26 P 3040bbe94988429cbc926e369d1f116a [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:53:30.195318 29313 tablet_replica.cc:330] T 911fdbb6a14748a2ab1424c496d43a26 P d2d22fd6ae0146a0a07ae3706506481a: stopping tablet replica
I20260928 08:53:30.195379 29315 raft_consensus.cc:2258] T 911fdbb6a14748a2ab1424c496d43a26 P 3040bbe94988429cbc926e369d1f116a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:53:30.195417 29313 raft_consensus.cc:2229] T 911fdbb6a14748a2ab1424c496d43a26 P d2d22fd6ae0146a0a07ae3706506481a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:53:30.195544 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 28886
I20260928 08:53:30.195552 29313 raft_consensus.cc:2258] T 911fdbb6a14748a2ab1424c496d43a26 P d2d22fd6ae0146a0a07ae3706506481a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:53:30.196663 29313 ts_tablet_manager.cc:1905] T 911fdbb6a14748a2ab1424c496d43a26 P d2d22fd6ae0146a0a07ae3706506481a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:53:30.198333 29313 ts_tablet_manager.cc:1918] T 911fdbb6a14748a2ab1424c496d43a26 P d2d22fd6ae0146a0a07ae3706506481a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:53:30.198411 29313 log.cc:1196] T 911fdbb6a14748a2ab1424c496d43a26 P d2d22fd6ae0146a0a07ae3706506481a: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585589999956-24223-0/minicluster-data/ts-2/wal/wals/911fdbb6a14748a2ab1424c496d43a26
I20260928 08:53:30.198657 29313 ts_tablet_manager.cc:1939] T 911fdbb6a14748a2ab1424c496d43a26 P d2d22fd6ae0146a0a07ae3706506481a: Deleting consensus metadata
I20260928 08:53:30.199080 28811 catalog_manager.cc:4841] TS d2d22fd6ae0146a0a07ae3706506481a (127.23.167.195:45251): tablet 911fdbb6a14748a2ab1424c496d43a26 (table test-table [id=edea594686aa474497312ff822d50e3e]) successfully deleted
W20260928 08:53:30.200711 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.167.193:39481 (error 108)
I20260928 08:53:30.201253 29287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:30.201129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:39481 (local address 127.0.0.1:50624)
0928 08:53:30.201170 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:30.201217 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.193:39481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 08:53:30.201321 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 29017
I20260928 08:53:30.201504 29315 ts_tablet_manager.cc:1905] T 911fdbb6a14748a2ab1424c496d43a26 P 3040bbe94988429cbc926e369d1f116a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 08:53:30.205675 28810 catalog_manager.cc:4584] TS c6fc180925c541e18891d120ba0f9f72 (127.23.167.193:39481): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 911fdbb6a14748a2ab1424c496d43a26: Network error: Client connection negotiation failed: client connection to 127.23.167.193:39481: connect: Connection refused (error 111)
I20260928 08:53:30.208571 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 29148
I20260928 08:53:30.208676 29287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:30.208557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:35773 (local address 127.0.0.1:55256)
0928 08:53:30.208611 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:30.208644 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.194:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 08:53:30.212385 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 28795
2026-09-28T08:53:30Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (589 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 08:53:30.231299 24223 mini_oidc.cc:152] Starting JWKS server
I20260928 08:53:30.232523 24223 webserver.cc:460] Webserver started at https://127.0.0.1:46319/ using document root <none> and password file <none>
I20260928 08:53:30.232627 24223 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 08:53:30.232800 24223 webserver.cc:460] Webserver started at http://127.0.0.1:39427/ using document root <none> and password file <none>
2026-09-28T08:53:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:53:30Z Disabled control of system clock
I20260928 08:53:30.239976 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:37061
--webserver_interface=127.23.167.254
--webserver_port=0
--builtin_ntp_servers=127.23.167.212:36147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:37061
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46319/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585589999956-24223-0/testchainca.cert with env {}
W20260928 08:53:30.318786 29322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:30.319005 29322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:30.319037 29322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:30.320235 29322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:53:30.320282 29322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:30.320298 29322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:30.320314 29322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:53:30.320328 29322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 08:53:30.320344 29322 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 08:53:30.320360 29322 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46319/jwks.json
I20260928 08:53:30.322086 29322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:36147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:37061
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:37061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46319/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585589999956-24223-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:30.322332 29322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:30.322542 29322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:30.327118 29330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:30.327131 29328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:30.327127 29327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:30.327195 29322 server_base.cc:1034] running on GCE node
I20260928 08:53:30.327544 29322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:30.327798 29322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:30.328948 29322 hybrid_clock.cc:648] HybridClock initialized: now 1790585610328934 us; error 27 us; skew 500 ppm
I20260928 08:53:30.330139 29322 webserver.cc:460] Webserver started at http://127.23.167.254:39783/ using document root <none> and password file <none>
I20260928 08:53:30.330356 29322 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:30.330407 29322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:30.330528 29322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:30.331580 29322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "731561d822344eb3a1b68a0fb2f8d1b0"
format_stamp: "Formatted at 2026-09-28 08:53:30 on dist-test-slave-pbjp"
server_key: "8803be8afdab17ae6b6cc94b58b86462"
server_key_iv: "930bfbd379d0df11a6d2a49e95eb2617"
server_key_version: "encryptionkey@0"
I20260928 08:53:30.331918 29322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "731561d822344eb3a1b68a0fb2f8d1b0"
format_stamp: "Formatted at 2026-09-28 08:53:30 on dist-test-slave-pbjp"
server_key: "8803be8afdab17ae6b6cc94b58b86462"
server_key_iv: "930bfbd379d0df11a6d2a49e95eb2617"
server_key_version: "encryptionkey@0"
I20260928 08:53:30.333256 29322 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 08:53:30.334054 29336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:30.334190 29322 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:53:30.334270 29322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "731561d822344eb3a1b68a0fb2f8d1b0"
format_stamp: "Formatted at 2026-09-28 08:53:30 on dist-test-slave-pbjp"
server_key: "8803be8afdab17ae6b6cc94b58b86462"
server_key_iv: "930bfbd379d0df11a6d2a49e95eb2617"
server_key_version: "encryptionkey@0"
I20260928 08:53:30.334337 29322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:30.356385 29322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:30.356717 29322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:30.356868 29322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:30.360512 29322 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:37061
I20260928 08:53:30.360569 29388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:37061 every 8 connection(s)
I20260928 08:53:30.360886 29322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:53:30.361833 29389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:30.364027 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 29322
I20260928 08:53:30.364130 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:53:30.364229 29389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 731561d822344eb3a1b68a0fb2f8d1b0: Bootstrap starting.
I20260928 08:53:30.364308 24223 external_mini_cluster.cc:1428] Setting key a22994a0d7813d844146e36172924e48
I20260928 08:53:30.364939 29389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 731561d822344eb3a1b68a0fb2f8d1b0: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:30.365249 29389 log.cc:824] T 00000000000000000000000000000000 P 731561d822344eb3a1b68a0fb2f8d1b0: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:30.365979 29389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 731561d822344eb3a1b68a0fb2f8d1b0: No bootstrap required, opened a new log
I20260928 08:53:30.368000 29389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 731561d822344eb3a1b68a0fb2f8d1b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731561d822344eb3a1b68a0fb2f8d1b0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 37061 } }
I20260928 08:53:30.368150 29389 raft_consensus.cc:374] T 00000000000000000000000000000000 P 731561d822344eb3a1b68a0fb2f8d1b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:30.368172 29389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 731561d822344eb3a1b68a0fb2f8d1b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731561d822344eb3a1b68a0fb2f8d1b0, State: Initialized, Role: FOLLOWER
I20260928 08:53:30.368284 29389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 731561d822344eb3a1b68a0fb2f8d1b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731561d822344eb3a1b68a0fb2f8d1b0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 37061 } }
I20260928 08:53:30.368351 29389 raft_consensus.cc:388] T 00000000000000000000000000000000 P 731561d822344eb3a1b68a0fb2f8d1b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:53:30.368384 29389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 731561d822344eb3a1b68a0fb2f8d1b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:53:30.368425 29389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 731561d822344eb3a1b68a0fb2f8d1b0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:30.369095 29389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 731561d822344eb3a1b68a0fb2f8d1b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731561d822344eb3a1b68a0fb2f8d1b0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 37061 } }
I20260928 08:53:30.369204 29389 leader_election.cc:302] T 00000000000000000000000000000000 P 731561d822344eb3a1b68a0fb2f8d1b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 731561d822344eb3a1b68a0fb2f8d1b0; no voters:
I20260928 08:53:30.369349 29389 leader_election.cc:288] T 00000000000000000000000000000000 P 731561d822344eb3a1b68a0fb2f8d1b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:53:30.369414 29394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 731561d822344eb3a1b68a0fb2f8d1b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:53:30.369583 29394 raft_consensus.cc:686] T 00000000000000000000000000000000 P 731561d822344eb3a1b68a0fb2f8d1b0 [term 1 LEADER]: Becoming Leader. State: Replica: 731561d822344eb3a1b68a0fb2f8d1b0, State: Running, Role: LEADER
I20260928 08:53:30.369630 29389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 731561d822344eb3a1b68a0fb2f8d1b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:53:30.369721 29394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 731561d822344eb3a1b68a0fb2f8d1b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731561d822344eb3a1b68a0fb2f8d1b0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 37061 } }
I20260928 08:53:30.370157 29392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:30.364802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43112 (local address 127.23.167.254:37061)
0928 08:53:30.365018 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:30.365027 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:30.365057 (+ 30us) server_negotiation.cc:407] Connection header received
0928 08:53:30.365545 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:30.365552 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:30.365614 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:30.365728 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:30.365948 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:30.366867 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:30.367986 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:30.368537 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:30.368647 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:30.369304 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:30.369309 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:30.369309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:53:30.369526 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:30.369572 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:53:30.369588 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:30.369845 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:30.369877 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 08:53:30.369960 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260928 08:53:30.370164 29396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 731561d822344eb3a1b68a0fb2f8d1b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 731561d822344eb3a1b68a0fb2f8d1b0. Latest consensus state: current_term: 1 leader_uuid: "731561d822344eb3a1b68a0fb2f8d1b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731561d822344eb3a1b68a0fb2f8d1b0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 37061 } } }
I20260928 08:53:30.370194 29395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 731561d822344eb3a1b68a0fb2f8d1b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "731561d822344eb3a1b68a0fb2f8d1b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731561d822344eb3a1b68a0fb2f8d1b0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 37061 } } }
I20260928 08:53:30.370271 29396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 731561d822344eb3a1b68a0fb2f8d1b0 [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:30.370296 29395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 731561d822344eb3a1b68a0fb2f8d1b0 [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:30.370817 29403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:53:30.371258 24223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:53:30.371335 29403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:53:30.372948 29403 catalog_manager.cc:1347] Generated new cluster ID: b80e28d99fe848a89f10ff815acebf34
I20260928 08:53:30.373008 29403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:53:30.397629 29403 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:53:30.398370 29403 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:53:30.402361 29403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 731561d822344eb3a1b68a0fb2f8d1b0: Generated new TSK 0
I20260928 08:53:30.402558 29403 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 08:53:30.423362 29419 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 08:53:30.423475 29419 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 08:53:30.423718 29392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:30.408968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43124 (local address 127.23.167.254:37061)
0928 08:53:30.409022 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:30.409025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:30.409105 (+ 80us) server_negotiation.cc:407] Connection header received
0928 08:53:30.409217 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:30.409218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:30.409229 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:30.409283 (+ 54us) server_negotiation.cc:226] Negotiated authn=JWT
0928 08:53:30.409416 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:30.409914 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:30.410654 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:30.410858 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:30.410911 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:30.411102 (+ 191us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 08:53:30.423475 (+ 12373us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46319/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 08:53:30.423647 (+ 172us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43124: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46319/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":32}
W20260928 08:53:30.423799 29392 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43124: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46319/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 08:53:30.424134 29418 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:53:30.408876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:37061 (local address 127.0.0.1:43124)
0928 08:53:30.409029 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:30.409039 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:30.409128 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:30.409289 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:30.409290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:30.409291 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0928 08:53:30.409361 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:30.409363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:30.409991 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:30.409992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:30.410579 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:30.410581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:30.410921 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:30.410922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:30.410984 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:30.411006 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 08:53:30.423623 (+ 12617us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46319/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 08:53:30.423778 (+ 155us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.167.254:37061: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46319/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":64,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.23.167.254:37061: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46319/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 08:53:30.424566 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 29322
2026-09-28T08:53:30Z chronyd exiting
I20260928 08:53:30.460147 24223 test_util.cc:175] -----------------------------------------------
I20260928 08:53:30.460238 24223 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585589999956-24223-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (229 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 08:53:30.461018 24223 mini_oidc.cc:152] Starting JWKS server
I20260928 08:53:30.461760 24223 webserver.cc:460] Webserver started at https://127.0.0.1:40935/ using document root <none> and password file <none>
I20260928 08:53:30.461805 24223 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 08:53:30.461951 24223 webserver.cc:460] Webserver started at http://127.0.0.1:38731/ using document root <none> and password file <none>
2026-09-28T08:53:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:53:30Z Disabled control of system clock
I20260928 08:53:30.468878 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:45795
--webserver_interface=127.23.167.254
--webserver_port=0
--builtin_ntp_servers=127.23.167.212:46409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:45795
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40935/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585589999956-24223-0/testchainca.cert with env {}
W20260928 08:53:30.548713 29425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:30.548921 29425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:30.548941 29425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:30.550064 29425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:53:30.550100 29425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:30.550112 29425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:30.550122 29425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:53:30.550133 29425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 08:53:30.550144 29425 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 08:53:30.550154 29425 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40935/jwks.json
I20260928 08:53:30.551615 29425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:46409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:45795
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:45795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40935/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585589999956-24223-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:30.551812 29425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:30.551980 29425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:30.556591 29430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:30.556629 29431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:30.556627 29425 server_base.cc:1034] running on GCE node
W20260928 08:53:30.556605 29433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:30.556954 29425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:30.557269 29425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:30.558427 29425 hybrid_clock.cc:648] HybridClock initialized: now 1790585610558410 us; error 32 us; skew 500 ppm
I20260928 08:53:30.559628 29425 webserver.cc:460] Webserver started at http://127.23.167.254:35063/ using document root <none> and password file <none>
I20260928 08:53:30.559845 29425 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:30.559890 29425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:30.560019 29425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:30.561165 29425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "f0ff9aa0d73b4a13ae6d2c87d40d1dba"
format_stamp: "Formatted at 2026-09-28 08:53:30 on dist-test-slave-pbjp"
server_key: "fd6e97bbac5c45387101160c822536a0"
server_key_iv: "10bb684adec6d520cc2ded071830c30b"
server_key_version: "encryptionkey@0"
I20260928 08:53:30.561501 29425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "f0ff9aa0d73b4a13ae6d2c87d40d1dba"
format_stamp: "Formatted at 2026-09-28 08:53:30 on dist-test-slave-pbjp"
server_key: "fd6e97bbac5c45387101160c822536a0"
server_key_iv: "10bb684adec6d520cc2ded071830c30b"
server_key_version: "encryptionkey@0"
I20260928 08:53:30.562675 29425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:53:30.563305 29439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:30.563455 29425 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:53:30.563537 29425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "f0ff9aa0d73b4a13ae6d2c87d40d1dba"
format_stamp: "Formatted at 2026-09-28 08:53:30 on dist-test-slave-pbjp"
server_key: "fd6e97bbac5c45387101160c822536a0"
server_key_iv: "10bb684adec6d520cc2ded071830c30b"
server_key_version: "encryptionkey@0"
I20260928 08:53:30.563598 29425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:30.571460 29425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:30.571741 29425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:30.571870 29425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:30.575451 29425 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:45795
I20260928 08:53:30.575481 29491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:45795 every 8 connection(s)
I20260928 08:53:30.575781 29425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:53:30.576392 29492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:30.579190 29492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0ff9aa0d73b4a13ae6d2c87d40d1dba: Bootstrap starting.
I20260928 08:53:30.579715 29492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0ff9aa0d73b4a13ae6d2c87d40d1dba: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:30.579932 29492 log.cc:824] T 00000000000000000000000000000000 P f0ff9aa0d73b4a13ae6d2c87d40d1dba: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:30.580554 29492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0ff9aa0d73b4a13ae6d2c87d40d1dba: No bootstrap required, opened a new log
I20260928 08:53:30.582125 29492 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0ff9aa0d73b4a13ae6d2c87d40d1dba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ff9aa0d73b4a13ae6d2c87d40d1dba" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 45795 } }
I20260928 08:53:30.582259 29492 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0ff9aa0d73b4a13ae6d2c87d40d1dba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:30.582280 29492 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0ff9aa0d73b4a13ae6d2c87d40d1dba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0ff9aa0d73b4a13ae6d2c87d40d1dba, State: Initialized, Role: FOLLOWER
I20260928 08:53:30.582389 29492 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0ff9aa0d73b4a13ae6d2c87d40d1dba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ff9aa0d73b4a13ae6d2c87d40d1dba" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 45795 } }
I20260928 08:53:30.582461 29492 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0ff9aa0d73b4a13ae6d2c87d40d1dba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:53:30.582505 29492 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0ff9aa0d73b4a13ae6d2c87d40d1dba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:53:30.582554 29492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0ff9aa0d73b4a13ae6d2c87d40d1dba [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:30.582649 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 29425
I20260928 08:53:30.582742 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:53:30.582929 24223 external_mini_cluster.cc:1428] Setting key d744bd9186766f125b2b3c26a80f1c8a
I20260928 08:53:30.583197 29492 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0ff9aa0d73b4a13ae6d2c87d40d1dba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ff9aa0d73b4a13ae6d2c87d40d1dba" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 45795 } }
I20260928 08:53:30.583302 29492 leader_election.cc:302] T 00000000000000000000000000000000 P f0ff9aa0d73b4a13ae6d2c87d40d1dba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0ff9aa0d73b4a13ae6d2c87d40d1dba; no voters:
I20260928 08:53:30.583467 29492 leader_election.cc:288] T 00000000000000000000000000000000 P f0ff9aa0d73b4a13ae6d2c87d40d1dba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:53:30.583554 29496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0ff9aa0d73b4a13ae6d2c87d40d1dba [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:53:30.583654 29496 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0ff9aa0d73b4a13ae6d2c87d40d1dba [term 1 LEADER]: Becoming Leader. State: Replica: f0ff9aa0d73b4a13ae6d2c87d40d1dba, State: Running, Role: LEADER
I20260928 08:53:30.583730 29492 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0ff9aa0d73b4a13ae6d2c87d40d1dba [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:53:30.583731 29496 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0ff9aa0d73b4a13ae6d2c87d40d1dba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ff9aa0d73b4a13ae6d2c87d40d1dba" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 45795 } }
I20260928 08:53:30.584148 29498 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0ff9aa0d73b4a13ae6d2c87d40d1dba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0ff9aa0d73b4a13ae6d2c87d40d1dba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ff9aa0d73b4a13ae6d2c87d40d1dba" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 45795 } } }
I20260928 08:53:30.584226 29498 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0ff9aa0d73b4a13ae6d2c87d40d1dba [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:30.584424 29503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:53:30.584563 29499 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0ff9aa0d73b4a13ae6d2c87d40d1dba [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0ff9aa0d73b4a13ae6d2c87d40d1dba. Latest consensus state: current_term: 1 leader_uuid: "f0ff9aa0d73b4a13ae6d2c87d40d1dba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ff9aa0d73b4a13ae6d2c87d40d1dba" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 45795 } } }
I20260928 08:53:30.584630 29499 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0ff9aa0d73b4a13ae6d2c87d40d1dba [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:30.585049 29503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:53:30.586947 29503 catalog_manager.cc:1347] Generated new cluster ID: 7e322baa63ad4ccea3166f6eb50997f9
I20260928 08:53:30.586997 29503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:53:30.588775 29497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:30.583362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60370 (local address 127.23.167.254:45795)
0928 08:53:30.583659 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:30.583668 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:30.583753 (+ 85us) server_negotiation.cc:407] Connection header received
0928 08:53:30.583872 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:30.583879 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:30.583958 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:30.584089 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:30.584300 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:30.585685 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:30.586702 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:30.587280 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:30.587408 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:30.588051 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:30.588055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:30.588055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:53:30.588258 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:30.588300 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:53:30.588314 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:30.588386 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:30.588444 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 08:53:30.588525 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":73,"threads_started":1}
I20260928 08:53:30.589697 24223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:53:30.594990 29503 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:53:30.595674 29503 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:53:30.598616 29503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0ff9aa0d73b4a13ae6d2c87d40d1dba: Generated new TSK 0
I20260928 08:53:30.598788 29503 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 08:53:30.621973 29522 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 08:53:30.622051 29522 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 08:53:30.622433 29497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:30.607587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60384 (local address 127.23.167.254:45795)
0928 08:53:30.607630 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:30.607632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:30.607734 (+ 102us) server_negotiation.cc:407] Connection header received
0928 08:53:30.607805 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:30.607806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:30.607820 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:30.607908 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0928 08:53:30.608036 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:30.608830 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:30.609660 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:30.610003 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:30.610074 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:30.610218 (+ 144us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 08:53:30.622193 (+ 11975us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40935/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 08:53:30.622356 (+ 163us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60384: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40935/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 08:53:30.622498 29497 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60384: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40935/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 08:53:30.622520 29521 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:53:30.607520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:45795 (local address 127.0.0.1:60384)
0928 08:53:30.607660 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:30.607671 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:30.607731 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:30.607911 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:30.607912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:30.607914 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 08:53:30.607978 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:30.607980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:30.608923 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:30.608924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:30.609576 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:30.609580 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:30.610079 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:30.610080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:30.610127 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:30.610133 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 08:53:30.622321 (+ 12188us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40935/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 08:53:30.622435 (+ 114us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.167.254:45795: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40935/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260928 08:53:30.622792 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 29425
2026-09-28T08:53:30Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (198 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 08:53:30.659735 24223 mini_oidc.cc:152] Starting JWKS server
I20260928 08:53:30.660413 24223 webserver.cc:460] Webserver started at https://127.0.0.1:34469/ using document root <none> and password file <none>
I20260928 08:53:30.660456 24223 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 08:53:30.660566 24223 webserver.cc:460] Webserver started at http://127.0.0.1:42391/ using document root <none> and password file <none>
2026-09-28T08:53:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:53:30Z Disabled control of system clock
I20260928 08:53:30.667709 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:41077
--webserver_interface=127.23.167.254
--webserver_port=0
--builtin_ntp_servers=127.23.167.212:42317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:41077
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34469/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 08:53:30.747594 29528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:30.747828 29528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:30.747859 29528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:30.749151 29528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:53:30.749199 29528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:30.749215 29528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:30.749233 29528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:53:30.749248 29528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 08:53:30.749264 29528 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 08:53:30.749284 29528 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34469/jwks.json
I20260928 08:53:30.750737 29528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:42317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:41077
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:41077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34469/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:30.750990 29528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:30.751292 29528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:30.756089 29533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:30.756166 29528 server_base.cc:1034] running on GCE node
W20260928 08:53:30.756067 29536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:30.756069 29534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:30.756556 29528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:30.756916 29528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:30.758069 29528 hybrid_clock.cc:648] HybridClock initialized: now 1790585610758049 us; error 44 us; skew 500 ppm
I20260928 08:53:30.759624 29528 webserver.cc:460] Webserver started at http://127.23.167.254:39707/ using document root <none> and password file <none>
I20260928 08:53:30.759920 29528 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:30.759989 29528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:30.760138 29528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:30.761301 29528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "0350203a064647ce8fa2e5ea72b8f455"
format_stamp: "Formatted at 2026-09-28 08:53:30 on dist-test-slave-pbjp"
server_key: "4330061b52e1ae97ac7a48197aad35b4"
server_key_iv: "9f95852e86ac2d58d82f5b91c1d3abed"
server_key_version: "encryptionkey@0"
I20260928 08:53:30.761638 29528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "0350203a064647ce8fa2e5ea72b8f455"
format_stamp: "Formatted at 2026-09-28 08:53:30 on dist-test-slave-pbjp"
server_key: "4330061b52e1ae97ac7a48197aad35b4"
server_key_iv: "9f95852e86ac2d58d82f5b91c1d3abed"
server_key_version: "encryptionkey@0"
I20260928 08:53:30.762904 29528 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:53:30.763638 29542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:30.763809 29528 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:53:30.763887 29528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "0350203a064647ce8fa2e5ea72b8f455"
format_stamp: "Formatted at 2026-09-28 08:53:30 on dist-test-slave-pbjp"
server_key: "4330061b52e1ae97ac7a48197aad35b4"
server_key_iv: "9f95852e86ac2d58d82f5b91c1d3abed"
server_key_version: "encryptionkey@0"
I20260928 08:53:30.763942 29528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:30.783048 29528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:30.783361 29528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:30.783468 29528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:30.787139 29528 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:41077
I20260928 08:53:30.787178 29594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:41077 every 8 connection(s)
I20260928 08:53:30.787518 29528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:53:30.788190 29595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:30.790942 29595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0350203a064647ce8fa2e5ea72b8f455: Bootstrap starting.
I20260928 08:53:30.791489 29595 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0350203a064647ce8fa2e5ea72b8f455: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:30.791746 29595 log.cc:824] T 00000000000000000000000000000000 P 0350203a064647ce8fa2e5ea72b8f455: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:30.791798 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 29528
I20260928 08:53:30.791878 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:53:30.792050 24223 external_mini_cluster.cc:1428] Setting key 691a2c3178cb84bd8650623350871f9e
I20260928 08:53:30.792483 29595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0350203a064647ce8fa2e5ea72b8f455: No bootstrap required, opened a new log
I20260928 08:53:30.795243 29595 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0350203a064647ce8fa2e5ea72b8f455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0350203a064647ce8fa2e5ea72b8f455" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 41077 } }
I20260928 08:53:30.795413 29595 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0350203a064647ce8fa2e5ea72b8f455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:30.795442 29595 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0350203a064647ce8fa2e5ea72b8f455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0350203a064647ce8fa2e5ea72b8f455, State: Initialized, Role: FOLLOWER
I20260928 08:53:30.795557 29595 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0350203a064647ce8fa2e5ea72b8f455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0350203a064647ce8fa2e5ea72b8f455" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 41077 } }
I20260928 08:53:30.795625 29595 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0350203a064647ce8fa2e5ea72b8f455 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:53:30.795661 29595 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0350203a064647ce8fa2e5ea72b8f455 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:53:30.795713 29595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0350203a064647ce8fa2e5ea72b8f455 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:30.796262 29595 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0350203a064647ce8fa2e5ea72b8f455 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0350203a064647ce8fa2e5ea72b8f455" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 41077 } }
I20260928 08:53:30.796350 29595 leader_election.cc:302] T 00000000000000000000000000000000 P 0350203a064647ce8fa2e5ea72b8f455 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0350203a064647ce8fa2e5ea72b8f455; no voters:
I20260928 08:53:30.796512 29595 leader_election.cc:288] T 00000000000000000000000000000000 P 0350203a064647ce8fa2e5ea72b8f455 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:53:30.796553 29600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0350203a064647ce8fa2e5ea72b8f455 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:53:30.796666 29600 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0350203a064647ce8fa2e5ea72b8f455 [term 1 LEADER]: Becoming Leader. State: Replica: 0350203a064647ce8fa2e5ea72b8f455, State: Running, Role: LEADER
I20260928 08:53:30.796746 29600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0350203a064647ce8fa2e5ea72b8f455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0350203a064647ce8fa2e5ea72b8f455" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 41077 } }
I20260928 08:53:30.796770 29595 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0350203a064647ce8fa2e5ea72b8f455 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:53:30.797171 29602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0350203a064647ce8fa2e5ea72b8f455 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0350203a064647ce8fa2e5ea72b8f455" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0350203a064647ce8fa2e5ea72b8f455" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 41077 } } }
I20260928 08:53:30.797246 29602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0350203a064647ce8fa2e5ea72b8f455 [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:30.797412 29601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0350203a064647ce8fa2e5ea72b8f455 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0350203a064647ce8fa2e5ea72b8f455. Latest consensus state: current_term: 1 leader_uuid: "0350203a064647ce8fa2e5ea72b8f455" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0350203a064647ce8fa2e5ea72b8f455" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 41077 } } }
I20260928 08:53:30.797463 29601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0350203a064647ce8fa2e5ea72b8f455 [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:30.798048 29605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:53:30.798540 29605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:53:30.798761 29599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:30.792583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54234 (local address 127.23.167.254:41077)
0928 08:53:30.792792 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:30.792801 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:30.792836 (+ 35us) server_negotiation.cc:407] Connection header received
0928 08:53:30.793807 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:30.793816 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:30.793898 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:30.794058 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:30.794256 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:30.795630 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:30.796469 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:30.797125 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:30.797581 (+ 456us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:30.797958 (+ 377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:30.797963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:30.797963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:53:30.798172 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:30.798215 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:53:30.798230 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:30.798288 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:30.798479 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 08:53:30.798558 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
I20260928 08:53:30.799909 24223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:53:30.800129 29605 catalog_manager.cc:1347] Generated new cluster ID: be3b97b4c2124c308f74797736591753
I20260928 08:53:30.800173 29605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:53:30.814656 29605 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:53:30.815188 29605 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:53:30.821369 29605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0350203a064647ce8fa2e5ea72b8f455: Generated new TSK 0
I20260928 08:53:30.821557 29605 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:53:30.832214 29599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:30.828714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54250 (local address 127.23.167.254:41077)
0928 08:53:30.828764 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:30.828766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:30.828886 (+ 120us) server_negotiation.cc:407] Connection header received
0928 08:53:30.828982 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:30.828983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:30.828994 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:30.829090 (+ 96us) server_negotiation.cc:226] Negotiated authn=JWT
0928 08:53:30.829229 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:30.829725 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:30.830811 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:30.831000 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:30.831045 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:30.831205 (+ 160us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 08:53:30.832013 (+ 808us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34469/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 08:53:30.832141 (+ 128us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54250: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34469/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 08:53:30.832285 29599 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54250: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34469/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 08:53:30.832300 29624 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:53:30.828642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:41077 (local address 127.0.0.1:54250)
0928 08:53:30.828817 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:30.828828 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:30.828905 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:30.829104 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:30.829105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:30.829108 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0928 08:53:30.829169 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:30.829172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:30.829797 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:30.829798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:30.830736 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:30.830740 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:30.831060 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:30.831061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:30.831120 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:30.831126 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 08:53:30.832116 (+ 990us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34469/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 08:53:30.832205 (+ 89us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.167.254:41077: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34469/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":82,"threads_started":1}
I20260928 08:53:30.832436 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 29528
E20260928 08:53:30.844549 29625 webserver.cc:533] Webserver: sslize failed:
E20260928 08:53:30.844625 29625 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T08:53:30Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (210 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T08:53:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:53:30Z Disabled control of system clock
I20260928 08:53:30.877017 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:41931
--webserver_interface=127.23.167.254
--webserver_port=0
--builtin_ntp_servers=127.23.167.212:39735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:41931
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585589999956-24223-0/testchainca.cert with env {}
W20260928 08:53:30.956640 29629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:30.956861 29629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:30.956883 29629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:30.958112 29629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:53:30.958153 29629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:30.958165 29629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:30.958176 29629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:53:30.958187 29629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 08:53:30.958199 29629 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 08:53:30.958209 29629 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 08:53:30.959704 29629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:39735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:41931
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:41931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585589999956-24223-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:30.959913 29629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:30.960081 29629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:30.964735 29634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:30.964745 29635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:30.964732 29637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:30.964994 29629 server_base.cc:1034] running on GCE node
I20260928 08:53:30.965252 29629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:30.965538 29629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:30.966675 29629 hybrid_clock.cc:648] HybridClock initialized: now 1790585610966661 us; error 32 us; skew 500 ppm
I20260928 08:53:30.967955 29629 webserver.cc:460] Webserver started at http://127.23.167.254:36413/ using document root <none> and password file <none>
I20260928 08:53:30.968168 29629 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:30.968212 29629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:30.968328 29629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:30.969432 29629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "74b1753ef4874ae08928e0f9cfe08b6c"
format_stamp: "Formatted at 2026-09-28 08:53:30 on dist-test-slave-pbjp"
server_key: "6962c5da5a2c7bb343ef5e780bc74cf8"
server_key_iv: "85e9dfc53c9604f3f578e476b9a76281"
server_key_version: "encryptionkey@0"
I20260928 08:53:30.969767 29629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "74b1753ef4874ae08928e0f9cfe08b6c"
format_stamp: "Formatted at 2026-09-28 08:53:30 on dist-test-slave-pbjp"
server_key: "6962c5da5a2c7bb343ef5e780bc74cf8"
server_key_iv: "85e9dfc53c9604f3f578e476b9a76281"
server_key_version: "encryptionkey@0"
I20260928 08:53:30.971197 29629 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:53:30.971906 29643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:30.972050 29629 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:53:30.972132 29629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "74b1753ef4874ae08928e0f9cfe08b6c"
format_stamp: "Formatted at 2026-09-28 08:53:30 on dist-test-slave-pbjp"
server_key: "6962c5da5a2c7bb343ef5e780bc74cf8"
server_key_iv: "85e9dfc53c9604f3f578e476b9a76281"
server_key_version: "encryptionkey@0"
I20260928 08:53:30.972191 29629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:30.988312 29629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:30.988610 29629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:30.988729 29629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:30.992596 29629 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:41931
I20260928 08:53:30.992642 29695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:41931 every 8 connection(s)
I20260928 08:53:30.992961 29629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:53:30.993929 29696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:30.996134 29696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74b1753ef4874ae08928e0f9cfe08b6c: Bootstrap starting.
I20260928 08:53:30.996693 29696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74b1753ef4874ae08928e0f9cfe08b6c: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:30.996954 29696 log.cc:824] T 00000000000000000000000000000000 P 74b1753ef4874ae08928e0f9cfe08b6c: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:30.997646 29696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74b1753ef4874ae08928e0f9cfe08b6c: No bootstrap required, opened a new log
I20260928 08:53:30.999177 29696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74b1753ef4874ae08928e0f9cfe08b6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b1753ef4874ae08928e0f9cfe08b6c" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 41931 } }
I20260928 08:53:30.999329 29696 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74b1753ef4874ae08928e0f9cfe08b6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:30.999351 29696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74b1753ef4874ae08928e0f9cfe08b6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74b1753ef4874ae08928e0f9cfe08b6c, State: Initialized, Role: FOLLOWER
I20260928 08:53:30.999496 29696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74b1753ef4874ae08928e0f9cfe08b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b1753ef4874ae08928e0f9cfe08b6c" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 41931 } }
I20260928 08:53:30.999562 29696 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74b1753ef4874ae08928e0f9cfe08b6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:53:30.999589 29696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74b1753ef4874ae08928e0f9cfe08b6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:53:30.999634 29696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74b1753ef4874ae08928e0f9cfe08b6c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:31.000125 29696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74b1753ef4874ae08928e0f9cfe08b6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b1753ef4874ae08928e0f9cfe08b6c" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 41931 } }
I20260928 08:53:31.000216 29696 leader_election.cc:302] T 00000000000000000000000000000000 P 74b1753ef4874ae08928e0f9cfe08b6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74b1753ef4874ae08928e0f9cfe08b6c; no voters:
I20260928 08:53:31.000358 29696 leader_election.cc:288] T 00000000000000000000000000000000 P 74b1753ef4874ae08928e0f9cfe08b6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:53:31.000408 29699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74b1753ef4874ae08928e0f9cfe08b6c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:53:31.000567 29699 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74b1753ef4874ae08928e0f9cfe08b6c [term 1 LEADER]: Becoming Leader. State: Replica: 74b1753ef4874ae08928e0f9cfe08b6c, State: Running, Role: LEADER
I20260928 08:53:31.000599 29696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74b1753ef4874ae08928e0f9cfe08b6c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:53:31.000666 29699 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74b1753ef4874ae08928e0f9cfe08b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b1753ef4874ae08928e0f9cfe08b6c" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 41931 } }
I20260928 08:53:31.001086 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 29629
I20260928 08:53:31.001183 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:53:31.001359 24223 external_mini_cluster.cc:1428] Setting key 4348eff07006519969c5745221ed66d2
I20260928 08:53:31.001348 29701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74b1753ef4874ae08928e0f9cfe08b6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74b1753ef4874ae08928e0f9cfe08b6c. Latest consensus state: current_term: 1 leader_uuid: "74b1753ef4874ae08928e0f9cfe08b6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b1753ef4874ae08928e0f9cfe08b6c" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 41931 } } }
I20260928 08:53:31.001432 29701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74b1753ef4874ae08928e0f9cfe08b6c [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:31.001444 29700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74b1753ef4874ae08928e0f9cfe08b6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74b1753ef4874ae08928e0f9cfe08b6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b1753ef4874ae08928e0f9cfe08b6c" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 41931 } } }
I20260928 08:53:31.001500 29700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74b1753ef4874ae08928e0f9cfe08b6c [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:31.001641 29708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:53:31.002161 29708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:53:31.003628 29708 catalog_manager.cc:1347] Generated new cluster ID: 19e307748acf4220bf2fac15054e9709
I20260928 08:53:31.003677 29708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:53:31.007481 29711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:31.001808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45302 (local address 127.23.167.254:41931)
0928 08:53:31.001998 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:31.002007 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:31.002105 (+ 98us) server_negotiation.cc:407] Connection header received
0928 08:53:31.002269 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:31.002277 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:31.002350 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:31.002810 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:31.002901 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:31.004223 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:31.005320 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:31.005985 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:31.006106 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:31.006769 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:31.006773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:31.006774 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:53:31.006993 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:31.007036 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:53:31.007051 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:31.007120 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:31.007174 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 08:53:31.007254 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260928 08:53:31.008388 24223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:53:31.019224 29708 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:53:31.019778 29708 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:53:31.023697 29708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74b1753ef4874ae08928e0f9cfe08b6c: Generated new TSK 0
I20260928 08:53:31.023867 29708 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:53:31.081848 29711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:31.031553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45310 (local address 127.23.167.254:41931)
0928 08:53:31.031608 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:31.031610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:31.031706 (+ 96us) server_negotiation.cc:407] Connection header received
0928 08:53:31.031789 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:31.031790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:31.031801 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:31.031878 (+ 77us) server_negotiation.cc:226] Negotiated authn=JWT
0928 08:53:31.031999 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:31.032524 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:31.033348 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:31.033550 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:31.033604 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:31.033781 (+ 177us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 08:53:31.081575 (+ 47794us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 08:53:31.081769 (+ 194us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45310: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":30}
W20260928 08:53:31.081944 29711 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45310: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 08:53:31.081959 29725 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:53:31.031478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:41931 (local address 127.0.0.1:45310)
0928 08:53:31.031645 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:31.031655 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:31.031718 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:31.031882 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:31.031883 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:31.031885 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 08:53:31.031942 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:31.031944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:31.032609 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:31.032610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:31.033270 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:31.033273 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:31.033611 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:31.033612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:31.033670 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:31.033675 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 08:53:31.081737 (+ 48062us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 08:53:31.081858 (+ 121us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.167.254:41931: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":89,"threads_started":1}
I20260928 08:53:31.082202 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 29629
2026-09-28T08:53:31Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (228 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (87 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (86 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:53:31 dist-test-slave-pbjp krb5kdc[29737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:53:31 dist-test-slave-pbjp krb5kdc[29737](info): set up 2 sockets
Sep 28 08:53:31 dist-test-slave-pbjp krb5kdc[29737](info): commencing operation
krb5kdc: starting...
W20260928 08:53:32.204950 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:53:32 dist-test-slave-pbjp krb5kdc[29737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:53:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:53:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.167.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
I20260928 08:53:32.316290 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:46801
--webserver_interface=127.23.167.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:45145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:46801
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:32.396116 29750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:32.396327 29750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:32.396356 29750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:32.397567 29750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:53:32.397605 29750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:32.397624 29750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:32.397639 29750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:53:32.397655 29750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:53:32.399115 29750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:45145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:46801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:46801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:32.399367 29750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:32.399572 29750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:32.404366 29755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:32.404486 29750 server_base.cc:1034] running on GCE node
W20260928 08:53:32.404351 29758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:32.404351 29756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:32.404868 29750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:32.405192 29750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:32.406348 29750 hybrid_clock.cc:648] HybridClock initialized: now 1790585612406332 us; error 28 us; skew 500 ppm
Sep 28 08:53:32 dist-test-slave-pbjp krb5kdc[29737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585612, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:32.411649 29750 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:53:32.412097 29750 webserver.cc:460] Webserver started at http://127.23.167.254:42449/ using document root <none> and password file <none>
I20260928 08:53:32.412298 29750 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:32.412346 29750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:32.412477 29750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:32.413594 29750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "b3d55e97e3ba4f2ea25be01241a72ab7"
format_stamp: "Formatted at 2026-09-28 08:53:32 on dist-test-slave-pbjp"
server_key: "f1137db0ad3fe68b272d15423d3f0b53"
server_key_iv: "44022f9bd54d76f462eb3a382d575979"
server_key_version: "encryptionkey@0"
I20260928 08:53:32.413936 29750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "b3d55e97e3ba4f2ea25be01241a72ab7"
format_stamp: "Formatted at 2026-09-28 08:53:32 on dist-test-slave-pbjp"
server_key: "f1137db0ad3fe68b272d15423d3f0b53"
server_key_iv: "44022f9bd54d76f462eb3a382d575979"
server_key_version: "encryptionkey@0"
I20260928 08:53:32.415215 29750 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:53:32.415891 29765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:32.416045 29750 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:53:32.416132 29750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "b3d55e97e3ba4f2ea25be01241a72ab7"
format_stamp: "Formatted at 2026-09-28 08:53:32 on dist-test-slave-pbjp"
server_key: "f1137db0ad3fe68b272d15423d3f0b53"
server_key_iv: "44022f9bd54d76f462eb3a382d575979"
server_key_version: "encryptionkey@0"
I20260928 08:53:32.416191 29750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:32.430876 29750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:32.432639 29750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:32.432782 29750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:32.436559 29750 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:46801
I20260928 08:53:32.436623 29817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:46801 every 8 connection(s)
I20260928 08:53:32.436964 29750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:53:32.437852 29818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:32.440235 29818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3d55e97e3ba4f2ea25be01241a72ab7: Bootstrap starting.
I20260928 08:53:32.440830 29818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3d55e97e3ba4f2ea25be01241a72ab7: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:32.441035 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 29750
I20260928 08:53:32.441166 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:53:32.441169 29818 log.cc:824] T 00000000000000000000000000000000 P b3d55e97e3ba4f2ea25be01241a72ab7: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:32.441344 24223 external_mini_cluster.cc:1428] Setting key db39579a8715cca10d073f6817152179
I20260928 08:53:32.441908 29818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3d55e97e3ba4f2ea25be01241a72ab7: No bootstrap required, opened a new log
I20260928 08:53:32.443774 29818 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3d55e97e3ba4f2ea25be01241a72ab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3d55e97e3ba4f2ea25be01241a72ab7" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46801 } }
I20260928 08:53:32.443925 29818 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3d55e97e3ba4f2ea25be01241a72ab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:32.443959 29818 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3d55e97e3ba4f2ea25be01241a72ab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3d55e97e3ba4f2ea25be01241a72ab7, State: Initialized, Role: FOLLOWER
I20260928 08:53:32.444090 29818 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3d55e97e3ba4f2ea25be01241a72ab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3d55e97e3ba4f2ea25be01241a72ab7" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46801 } }
I20260928 08:53:32.444154 29818 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3d55e97e3ba4f2ea25be01241a72ab7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:53:32.444185 29818 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3d55e97e3ba4f2ea25be01241a72ab7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:53:32.444231 29818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3d55e97e3ba4f2ea25be01241a72ab7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:32.444811 29818 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3d55e97e3ba4f2ea25be01241a72ab7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3d55e97e3ba4f2ea25be01241a72ab7" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46801 } }
I20260928 08:53:32.444912 29818 leader_election.cc:302] T 00000000000000000000000000000000 P b3d55e97e3ba4f2ea25be01241a72ab7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3d55e97e3ba4f2ea25be01241a72ab7; no voters:
I20260928 08:53:32.445103 29818 leader_election.cc:288] T 00000000000000000000000000000000 P b3d55e97e3ba4f2ea25be01241a72ab7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:53:32.445129 29823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3d55e97e3ba4f2ea25be01241a72ab7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:53:32.445238 29823 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3d55e97e3ba4f2ea25be01241a72ab7 [term 1 LEADER]: Becoming Leader. State: Replica: b3d55e97e3ba4f2ea25be01241a72ab7, State: Running, Role: LEADER
I20260928 08:53:32.445333 29823 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3d55e97e3ba4f2ea25be01241a72ab7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3d55e97e3ba4f2ea25be01241a72ab7" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46801 } }
I20260928 08:53:32.445391 29818 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3d55e97e3ba4f2ea25be01241a72ab7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:53:32.445727 29823 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3d55e97e3ba4f2ea25be01241a72ab7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3d55e97e3ba4f2ea25be01241a72ab7. Latest consensus state: current_term: 1 leader_uuid: "b3d55e97e3ba4f2ea25be01241a72ab7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3d55e97e3ba4f2ea25be01241a72ab7" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46801 } } }
I20260928 08:53:32.445735 29824 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3d55e97e3ba4f2ea25be01241a72ab7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3d55e97e3ba4f2ea25be01241a72ab7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3d55e97e3ba4f2ea25be01241a72ab7" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46801 } } }
I20260928 08:53:32.445811 29823 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3d55e97e3ba4f2ea25be01241a72ab7 [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:32.445833 29824 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3d55e97e3ba4f2ea25be01241a72ab7 [sys.catalog]: This master's current role is: LEADER
W20260928 08:53:32.447158 29839 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b3d55e97e3ba4f2ea25be01241a72ab7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:53:32.447211 29839 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:53:32.447254 29829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:53:32.447388 29829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:53:32.448872 29829 catalog_manager.cc:1347] Generated new cluster ID: cf504ae5f0a44c7d89e1436a849b3f2b
I20260928 08:53:32.448923 29829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 08:53:32 dist-test-slave-pbjp krb5kdc[29737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:32.452802 29822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:32.441828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42946 (local address 127.23.167.254:46801)
0928 08:53:32.442046 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:32.442054 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:32.442081 (+ 27us) server_negotiation.cc:407] Connection header received
0928 08:53:32.443094 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:32.443102 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:32.443157 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:32.443293 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:32.443949 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:32.445357 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:32.446434 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:32.447067 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:32.447165 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:32.449547 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:32.449552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:32.449552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:32.449565 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:32.451086 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:32.451409 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:32.451412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:32.451413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:32.451504 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:32.451677 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:32.451678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:32.451678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:32.451912 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:32.452068 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:32.452162 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 08:53:32.452403 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":80,"threads_started":1}
I20260928 08:53:32.459784 29829 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:53:32.460484 29829 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:53:32.475358 29829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3d55e97e3ba4f2ea25be01241a72ab7: Generated new TSK 0
I20260928 08:53:32.475581 29829 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.167.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
I20260928 08:53:32.502192 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:0
--local_ip_for_outbound_sockets=127.23.167.193
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:46801
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:45145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:32.584127 29844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:32.584334 29844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:32.584376 29844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:32.585552 29844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:32.585592 29844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:32.585626 29844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:53:32.587146 29844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:45145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:46801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:32.587400 29844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:32.587622 29844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:32.592521 29849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:32.592553 29852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:32.592517 29850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:32.592555 29844 server_base.cc:1034] running on GCE node
I20260928 08:53:32.592916 29844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:32.593164 29844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:32.594365 29844 hybrid_clock.cc:648] HybridClock initialized: now 1790585612594351 us; error 28 us; skew 500 ppm
Sep 28 08:53:32 dist-test-slave-pbjp krb5kdc[29737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585612, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:32.596854 29844 init.cc:377] Logged in from keytab as kudu/127.23.167.193@KRBTEST.COM (short username kudu)
I20260928 08:53:32.597368 29844 webserver.cc:460] Webserver started at http://127.23.167.193:42133/ using document root <none> and password file <none>
I20260928 08:53:32.597585 29844 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:32.597632 29844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:32.597751 29844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:32.598776 29844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-0/data/instance:
uuid: "a55f3a4aeeea453399afb9d412cf0b03"
format_stamp: "Formatted at 2026-09-28 08:53:32 on dist-test-slave-pbjp"
server_key: "9a0f45fe4388d6a6aa3f5b25b25ce3dd"
server_key_iv: "6f1e10126e745a4b5943e0cee833cea2"
server_key_version: "encryptionkey@0"
I20260928 08:53:32.599092 29844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance:
uuid: "a55f3a4aeeea453399afb9d412cf0b03"
format_stamp: "Formatted at 2026-09-28 08:53:32 on dist-test-slave-pbjp"
server_key: "9a0f45fe4388d6a6aa3f5b25b25ce3dd"
server_key_iv: "6f1e10126e745a4b5943e0cee833cea2"
server_key_version: "encryptionkey@0"
I20260928 08:53:32.600400 29844 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:53:32.601176 29859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:32.601356 29844 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:53:32.601444 29844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "a55f3a4aeeea453399afb9d412cf0b03"
format_stamp: "Formatted at 2026-09-28 08:53:32 on dist-test-slave-pbjp"
server_key: "9a0f45fe4388d6a6aa3f5b25b25ce3dd"
server_key_iv: "6f1e10126e745a4b5943e0cee833cea2"
server_key_version: "encryptionkey@0"
I20260928 08:53:32.601512 29844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:32.618870 29844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:32.621047 29844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:32.621222 29844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:32.621469 29844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:53:32.621767 29844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:32.621803 29844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:32.621833 29844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:32.621853 29844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:32.627349 29844 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:35919
I20260928 08:53:32.627403 29972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:35919 every 8 connection(s)
I20260928 08:53:32.627696 29844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
Sep 28 08:53:32 dist-test-slave-pbjp krb5kdc[29737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585612, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:32.636641 29822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:32.628419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:46169 (local address 127.23.167.254:46801)
0928 08:53:32.628488 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:32.628490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:32.628659 (+ 169us) server_negotiation.cc:407] Connection header received
0928 08:53:32.629565 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:32.629566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:32.629577 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:32.629643 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:32.630237 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:32.630739 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:32.631792 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:32.631978 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:32.632024 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:32.634060 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:32.634061 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:32.634061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:32.634068 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:32.635328 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:32.635723 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:32.635724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:32.635725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:32.635773 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:32.636025 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:32.636025 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:32.636026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:32.636135 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:32.636186 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:32.636504 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 08:53:32.636573 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 08:53:32.636804 29975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:32.628381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:46801 (local address 127.23.167.193:46169)
0928 08:53:32.628530 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:32.628573 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:32.629475 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:32.629706 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:32.629710 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:32.629973 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:32.630169 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:32.630173 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:32.630833 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:32.630834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:32.631717 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:32.631719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:32.632036 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:32.632037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:32.632138 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:32.632625 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:32.632633 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:32.633963 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:32.635427 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:32.635434 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:32.635436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:32.635653 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:32.635847 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:32.635849 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:32.635849 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:32.635966 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:32.636221 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:32.636235 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:32.636374 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 08:53:32.636478 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 08:53:32.636607 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
I20260928 08:53:32.637279 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 29844
I20260928 08:53:32.637378 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance
I20260928 08:53:32.637396 29973 heartbeater.cc:347] Connected to a master server at 127.23.167.254:46801
I20260928 08:53:32.637492 29973 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:32.637566 24223 external_mini_cluster.cc:1428] Setting key b0256fd469a2fc8c8015710f9876c9f7
I20260928 08:53:32.637686 29973 heartbeater.cc:511] Master 127.23.167.254:46801 requested a full tablet report, sending...
I20260928 08:53:32.638190 29781 ts_manager.cc:194] Registered new tserver with Master: a55f3a4aeeea453399afb9d412cf0b03 (127.23.167.193:35919)
I20260928 08:53:32.638819 29781 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.193@KRBTEST.COM'} at 127.23.167.193:46169
WARNING: no policy specified for kudu/127.23.167.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
I20260928 08:53:32.661540 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:0
--local_ip_for_outbound_sockets=127.23.167.194
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:46801
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:45145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:32.740744 29978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:32.740984 29978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:32.741019 29978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:32.742278 29978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:32.742329 29978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:32.742367 29978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:53:32.743896 29978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:45145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:46801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:32.744139 29978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:32.744381 29978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:32.749533 29986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:32.749511 29983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:32.749569 29978 server_base.cc:1034] running on GCE node
W20260928 08:53:32.749511 29984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:32.749970 29978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:32.750234 29978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:32.751395 29978 hybrid_clock.cc:648] HybridClock initialized: now 1790585612751377 us; error 28 us; skew 500 ppm
Sep 28 08:53:32 dist-test-slave-pbjp krb5kdc[29737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585612, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:32.753929 29978 init.cc:377] Logged in from keytab as kudu/127.23.167.194@KRBTEST.COM (short username kudu)
I20260928 08:53:32.754386 29978 webserver.cc:460] Webserver started at http://127.23.167.194:35025/ using document root <none> and password file <none>
I20260928 08:53:32.754590 29978 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:32.754639 29978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:32.754769 29978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:32.755879 29978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-1/data/instance:
uuid: "37f98aba12704ef797bc2e69d5251283"
format_stamp: "Formatted at 2026-09-28 08:53:32 on dist-test-slave-pbjp"
server_key: "1172713ec0b3f833e9dd8b098ed5eac5"
server_key_iv: "ba175a99f8d2856b59fa8283199535c3"
server_key_version: "encryptionkey@0"
I20260928 08:53:32.756208 29978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance:
uuid: "37f98aba12704ef797bc2e69d5251283"
format_stamp: "Formatted at 2026-09-28 08:53:32 on dist-test-slave-pbjp"
server_key: "1172713ec0b3f833e9dd8b098ed5eac5"
server_key_iv: "ba175a99f8d2856b59fa8283199535c3"
server_key_version: "encryptionkey@0"
I20260928 08:53:32.757540 29978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 08:53:32.758239 29993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:32.758394 29978 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:53:32.758481 29978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "37f98aba12704ef797bc2e69d5251283"
format_stamp: "Formatted at 2026-09-28 08:53:32 on dist-test-slave-pbjp"
server_key: "1172713ec0b3f833e9dd8b098ed5eac5"
server_key_iv: "ba175a99f8d2856b59fa8283199535c3"
server_key_version: "encryptionkey@0"
I20260928 08:53:32.758540 29978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:32.774104 29978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:32.776204 29978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:32.776340 29978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:32.776552 29978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:53:32.776844 29978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:32.776877 29978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:32.776906 29978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:32.776930 29978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:32.782418 29978 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:43097
I20260928 08:53:32.782459 30106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:43097 every 8 connection(s)
I20260928 08:53:32.782755 29978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
I20260928 08:53:32.786013 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 29978
I20260928 08:53:32.786095 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance
I20260928 08:53:32.786235 24223 external_mini_cluster.cc:1428] Setting key 3b585b14ea99d219c3f7a123a4ffc0ef
Sep 28 08:53:32 dist-test-slave-pbjp krb5kdc[29737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585612, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:32.791719 29822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:32.783410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:52525 (local address 127.23.167.254:46801)
0928 08:53:32.783459 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:32.783460 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:32.783647 (+ 187us) server_negotiation.cc:407] Connection header received
0928 08:53:32.784554 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:32.784555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:32.784565 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:32.784604 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:32.785198 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:32.785682 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:32.786859 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:32.787084 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:32.787144 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:32.789171 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:32.789172 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:32.789173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:32.789180 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:32.790532 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:32.790880 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:32.790881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:32.790881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:32.790939 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:32.791139 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:32.791140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:32.791140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:32.791274 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:32.791351 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:32.791551 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 08:53:32.791625 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 08:53:32.791806 30109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:32.783392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:46801 (local address 127.23.167.194:52525)
0928 08:53:32.783536 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:32.783577 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:32.784451 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:32.784666 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:32.784669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:32.784910 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:32.785139 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:32.785144 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:32.785750 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:32.785750 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:32.786783 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:32.786785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:32.787163 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:32.787164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:32.787299 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:32.787759 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:32.787768 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:32.789082 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:32.790604 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:32.790610 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:32.790612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:32.790813 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:32.790999 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:32.791000 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:32.791000 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:32.791088 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:32.791331 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:32.791333 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:32.791442 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 08:53:32.791522 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 08:53:32.791636 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260928 08:53:32.792243 30107 heartbeater.cc:347] Connected to a master server at 127.23.167.254:46801
I20260928 08:53:32.792335 30107 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:32.792500 30107 heartbeater.cc:511] Master 127.23.167.254:46801 requested a full tablet report, sending...
I20260928 08:53:32.792874 29781 ts_manager.cc:194] Registered new tserver with Master: 37f98aba12704ef797bc2e69d5251283 (127.23.167.194:43097)
I20260928 08:53:32.793396 29781 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.194@KRBTEST.COM'} at 127.23.167.194:52525
WARNING: no policy specified for kudu/127.23.167.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
I20260928 08:53:32.811071 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:0
--local_ip_for_outbound_sockets=127.23.167.195
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:46801
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:45145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:32.889559 30112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:32.889755 30112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:32.889777 30112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:32.890842 30112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:32.890874 30112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:32.890897 30112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:53:32.892271 30112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:45145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:46801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:32.892452 30112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:32.892628 30112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:32.897217 30120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:32.897233 30118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:32.897265 30112 server_base.cc:1034] running on GCE node
W20260928 08:53:32.897286 30117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:32.897578 30112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:32.897774 30112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:32.898905 30112 hybrid_clock.cc:648] HybridClock initialized: now 1790585612898898 us; error 32 us; skew 500 ppm
Sep 28 08:53:32 dist-test-slave-pbjp krb5kdc[29737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585612, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:32.901237 30112 init.cc:377] Logged in from keytab as kudu/127.23.167.195@KRBTEST.COM (short username kudu)
I20260928 08:53:32.901603 30112 webserver.cc:460] Webserver started at http://127.23.167.195:33749/ using document root <none> and password file <none>
I20260928 08:53:32.901770 30112 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:32.901808 30112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:32.901895 30112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:32.902868 30112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-2/data/instance:
uuid: "124a985f668f452e9e9cb8d0cd2ca00d"
format_stamp: "Formatted at 2026-09-28 08:53:32 on dist-test-slave-pbjp"
server_key: "72035e4e3019a3b3e949ed153a7c2c1e"
server_key_iv: "13047d29ae1e397f4d8538845bf0a94c"
server_key_version: "encryptionkey@0"
I20260928 08:53:32.903162 30112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance:
uuid: "124a985f668f452e9e9cb8d0cd2ca00d"
format_stamp: "Formatted at 2026-09-28 08:53:32 on dist-test-slave-pbjp"
server_key: "72035e4e3019a3b3e949ed153a7c2c1e"
server_key_iv: "13047d29ae1e397f4d8538845bf0a94c"
server_key_version: "encryptionkey@0"
I20260928 08:53:32.904356 30112 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:53:32.904965 30127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:32.905129 30112 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:53:32.905210 30112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "124a985f668f452e9e9cb8d0cd2ca00d"
format_stamp: "Formatted at 2026-09-28 08:53:32 on dist-test-slave-pbjp"
server_key: "72035e4e3019a3b3e949ed153a7c2c1e"
server_key_iv: "13047d29ae1e397f4d8538845bf0a94c"
server_key_version: "encryptionkey@0"
I20260928 08:53:32.905279 30112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:32.912949 30112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:32.914863 30112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:32.915016 30112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:32.915251 30112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:53:32.915519 30112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:32.915549 30112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:32.915571 30112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:32.915588 30112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:32.921248 30112 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:37439
I20260928 08:53:32.921307 30240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:37439 every 8 connection(s)
I20260928 08:53:32.921617 30112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
I20260928 08:53:32.925323 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 30112
I20260928 08:53:32.925395 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance
I20260928 08:53:32.925557 24223 external_mini_cluster.cc:1428] Setting key 582974641a338999c363c73f10560634
Sep 28 08:53:32 dist-test-slave-pbjp krb5kdc[29737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585612, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:53:32.930958 29822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:32.922377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:43735 (local address 127.23.167.254:46801)
0928 08:53:32.922454 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:32.922456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:32.922613 (+ 157us) server_negotiation.cc:407] Connection header received
0928 08:53:32.923596 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:32.923597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:32.923611 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:32.923666 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:32.924350 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:32.924855 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:32.926069 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:32.926341 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:32.926403 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:32.928502 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:32.928504 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:32.928504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:32.928511 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:32.929648 (+ 1137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:32.930013 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:32.930014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:32.930015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:32.930069 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:32.930293 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:32.930294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:32.930294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:32.930407 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:32.930465 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:32.930750 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 08:53:32.930855 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 08:53:32.931089 30243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:32.922293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:46801 (local address 127.23.167.195:43735)
0928 08:53:32.922466 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:32.922516 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:32.923480 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:32.923754 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:32.923758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:32.924038 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:32.924248 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:32.924252 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:32.924940 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:32.924941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:32.926008 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:32.926011 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:32.926409 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:32.926410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:32.926543 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:32.927039 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:32.927048 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:32.928407 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:32.929723 (+ 1316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:32.929730 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:32.929733 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:32.929947 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:32.930130 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:32.930131 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:32.930131 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:32.930235 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:32.930463 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:32.930470 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:32.930609 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 08:53:32.930709 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 08:53:32.930868 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
I20260928 08:53:32.931538 30241 heartbeater.cc:347] Connected to a master server at 127.23.167.254:46801
I20260928 08:53:32.931627 30241 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:32.931785 30241 heartbeater.cc:511] Master 127.23.167.254:46801 requested a full tablet report, sending...
I20260928 08:53:32.932123 29781 ts_manager.cc:194] Registered new tserver with Master: 124a985f668f452e9e9cb8d0cd2ca00d (127.23.167.195:37439)
I20260928 08:53:32.932466 29781 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.195@KRBTEST.COM'} at 127.23.167.195:43735
I20260928 08:53:32.937307 24223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/test-admin.keytab.
Sep 28 08:53:32 dist-test-slave-pbjp krb5kdc[29737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:32.953258 24223 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 08:53:32.954761 30248 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:53:32.954043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55618 (local address 127.23.167.193:35919)
0928 08:53:32.954225 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:32.954230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:32.954242 (+ 12us) server_negotiation.cc:407] Connection header received
0928 08:53:32.954268 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:32.954270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:32.954283 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:32.954347 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:32.954692 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55618: BlockingRecv error: recv got EOF from 127.0.0.1:55618 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":99,"threads_started":1}
W20260928 08:53:32.954949 24223 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/krb5cc' of type FILE
Sep 28 08:53:32 dist-test-slave-pbjp krb5kdc[29737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:32.955897 24223 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 08:53:32 dist-test-slave-pbjp krb5kdc[29737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:32.956754 24223 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 08:53:32 dist-test-slave-pbjp krb5kdc[29737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.193@KRBTEST.COM
I20260928 08:53:32.964221 30248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:32.957294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55634 (local address 127.23.167.193:35919)
0928 08:53:32.957345 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:32.957348 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:32.957373 (+ 25us) server_negotiation.cc:407] Connection header received
0928 08:53:32.957452 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:32.957453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:32.957464 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:32.957503 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:32.957888 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:32.958612 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:32.959328 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:32.959662 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:32.959723 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:32.961770 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:32.961779 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:32.961779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:32.961786 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:32.963198 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:32.963518 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:32.963521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:32.963521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:32.963568 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:32.963713 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:32.963713 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:32.963713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:32.963835 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:32.963946 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:32.964054 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 08:53:32.964117 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260928 08:53:32.965889 30248 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:53:32.965482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55644 (local address 127.23.167.193:35919)
0928 08:53:32.965518 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:32.965520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:32.965544 (+ 24us) server_negotiation.cc:407] Connection header received
0928 08:53:32.965589 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:32.965589 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:32.965597 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:32.965636 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:32.965846 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55644: BlockingRecv error: recv got EOF from 127.0.0.1:55644 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 08:53:32.966025 24223 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/krb5cc' of type FILE
Sep 28 08:53:32 dist-test-slave-pbjp krb5kdc[29737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:32.966904 24223 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 08:53:32 dist-test-slave-pbjp krb5kdc[29737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:32.967655 24223 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 08:53:32 dist-test-slave-pbjp krb5kdc[29737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.193@KRBTEST.COM
I20260928 08:53:32.973937 30248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:32.968100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55660 (local address 127.23.167.193:35919)
0928 08:53:32.968149 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:32.968151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:32.968196 (+ 45us) server_negotiation.cc:407] Connection header received
0928 08:53:32.968234 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:32.968235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:32.968243 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:32.968284 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:32.968584 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:32.969050 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:32.969699 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:32.969888 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:32.969931 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:32.971667 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:32.971669 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:32.971669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:32.971675 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:32.972896 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:32.973309 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:32.973310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:32.973311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:32.973363 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:32.973558 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:32.973559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:32.973560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:32.973659 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:32.973732 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:32.973809 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 08:53:32.973862 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260928 08:53:32.975225 30248 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:53:32.974822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55670 (local address 127.23.167.193:35919)
0928 08:53:32.974873 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:32.974875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:32.974882 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:53:32.974899 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:32.974900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:32.974908 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:32.974952 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:32.975179 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55670: BlockingRecv error: recv got EOF from 127.0.0.1:55670 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260928 08:53:32.975286 24223 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585589999956-24223-0/krb5kdc/krb5cc' of type FILE
Sep 28 08:53:32 dist-test-slave-pbjp krb5kdc[29737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:32.976177 24223 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 08:53:32 dist-test-slave-pbjp krb5kdc[29737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:32.976924 24223 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 08:53:32 dist-test-slave-pbjp krb5kdc[29737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.193@KRBTEST.COM
I20260928 08:53:32.983129 30248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:32.977403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55672 (local address 127.23.167.193:35919)
0928 08:53:32.977439 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:32.977441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:32.977491 (+ 50us) server_negotiation.cc:407] Connection header received
0928 08:53:32.977537 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:32.977538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:32.977545 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:32.977586 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:32.977901 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:32.978347 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:32.978996 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:32.979176 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:32.979226 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:32.981053 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:32.981054 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:32.981054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:32.981083 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:32.982218 (+ 1135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:32.982526 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:32.982527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:32.982527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:32.982583 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:32.982757 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:32.982758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:32.982759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:32.982861 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:32.982949 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:32.983014 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 08:53:32.983066 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 08:53:32.983713 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 29844
I20260928 08:53:32.988351 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 29978
I20260928 08:53:32.992056 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 30112
I20260928 08:53:32.995523 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 29750
2026-09-28T08:53:32Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1740 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:53:33 dist-test-slave-pbjp krb5kdc[30261](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:53:33 dist-test-slave-pbjp krb5kdc[30261](info): set up 2 sockets
Sep 28 08:53:33 dist-test-slave-pbjp krb5kdc[30261](info): commencing operation
krb5kdc: starting...
W20260928 08:53:34.038225 24223 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260928 08:53:34.242115 24223 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.204s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260928 08:53:34.441097 24223 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.199s user 0.000s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260928 08:53:34.659915 24223 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.219s user 0.001s sys 0.000s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/test-user.keytab.
W20260928 08:53:35.730391 24223 mini_kdc.cc:302] Time spent creating keytab for test-user: real 1.070s user 0.001s sys 0.001s
Sep 28 08:53:35 dist-test-slave-pbjp krb5kdc[30261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:53:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:53:35Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.167.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.167.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.167.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.254.keytab.
Entry for principal oryx/127.23.167.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.254.keytab.
W20260928 08:53:36.715474 24223 mini_kdc.cc:262] Time spent creating service keytab for oryx/127.23.167.254: real 0.965s user 0.000s sys 0.002s
I20260928 08:53:36.715826 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:34791
--webserver_interface=127.23.167.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.254.keytab
--principal=oryx/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:36571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:34791
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:36.798338 30274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:36.798549 30274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:36.798579 30274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:36.799732 30274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:53:36.799773 30274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:36.799791 30274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:36.799808 30274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:53:36.799822 30274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:53:36.801326 30274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:36571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:34791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.254.keytab
--principal=oryx/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:34791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:36.801579 30274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:36.801784 30274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:36.806529 30282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:36.806547 30280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:36.806555 30274 server_base.cc:1034] running on GCE node
W20260928 08:53:36.806562 30279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:36.806979 30274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:36.807237 30274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:36.808413 30274 hybrid_clock.cc:648] HybridClock initialized: now 1790585616808390 us; error 34 us; skew 500 ppm
Sep 28 08:53:36 dist-test-slave-pbjp krb5kdc[30261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585616, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:36.811421 30274 init.cc:377] Logged in from keytab as oryx/127.23.167.254@KRBTEST.COM (short username oryx)
I20260928 08:53:36.811993 30274 webserver.cc:460] Webserver started at http://127.23.167.254:46299/ using document root <none> and password file <none>
I20260928 08:53:36.812250 30274 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:36.812312 30274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:36.812444 30274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:36.813918 30274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "0b528077fc484f4fae48815cbca7e762"
format_stamp: "Formatted at 2026-09-28 08:53:36 on dist-test-slave-pbjp"
server_key: "54ad9733df60e9120f0e7ba7c4dae1d5"
server_key_iv: "cfadc6874e50bcee917939e2422be1e9"
server_key_version: "encryptionkey@0"
I20260928 08:53:36.814319 30274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "0b528077fc484f4fae48815cbca7e762"
format_stamp: "Formatted at 2026-09-28 08:53:36 on dist-test-slave-pbjp"
server_key: "54ad9733df60e9120f0e7ba7c4dae1d5"
server_key_iv: "cfadc6874e50bcee917939e2422be1e9"
server_key_version: "encryptionkey@0"
I20260928 08:53:36.816075 30274 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:53:36.817140 30289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:36.817315 30274 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:53:36.817414 30274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "0b528077fc484f4fae48815cbca7e762"
format_stamp: "Formatted at 2026-09-28 08:53:36 on dist-test-slave-pbjp"
server_key: "54ad9733df60e9120f0e7ba7c4dae1d5"
server_key_iv: "cfadc6874e50bcee917939e2422be1e9"
server_key_version: "encryptionkey@0"
I20260928 08:53:36.817497 30274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:36.843413 30274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:36.846572 30274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:36.846804 30274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:36.852252 30274 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:34791
I20260928 08:53:36.852322 30341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:34791 every 8 connection(s)
I20260928 08:53:36.852845 30274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:53:36.853947 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 30274
I20260928 08:53:36.854035 30342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:36.854053 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:53:36.854260 24223 external_mini_cluster.cc:1428] Setting key 7e87bd19f54ac3382524518deef0cbff
I20260928 08:53:36.857607 30342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762: Bootstrap starting.
I20260928 08:53:36.858206 30342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:36.858510 30342 log.cc:824] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:36.859326 30342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762: No bootstrap required, opened a new log
I20260928 08:53:36.861567 30342 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } }
I20260928 08:53:36.861759 30342 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:36.861799 30342 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b528077fc484f4fae48815cbca7e762, State: Initialized, Role: FOLLOWER
I20260928 08:53:36.861912 30342 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } }
I20260928 08:53:36.861990 30342 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:53:36.862022 30342 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:53:36.862051 30342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 08:53:36 dist-test-slave-pbjp krb5kdc[30261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM
I20260928 08:53:36.866372 30345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:36.854873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47672 (local address 127.23.167.254:34791)
0928 08:53:36.855036 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:36.855047 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:36.855086 (+ 39us) server_negotiation.cc:407] Connection header received
0928 08:53:36.855739 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:36.855750 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:36.855826 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:36.855993 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:36.856753 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:36.858263 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:36.859294 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:36.859978 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:36.860089 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:36.862893 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:36.862897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:36.862898 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:36.862912 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:36.864707 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:36.865050 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:36.865053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:36.865054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:36.865199 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:36.865378 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:36.865378 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:36.865379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:36.865697 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:36.865835 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:36.865976 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 08:53:36.866121 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
I20260928 08:53:36.912436 30342 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } }
I20260928 08:53:36.912657 30342 leader_election.cc:302] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b528077fc484f4fae48815cbca7e762; no voters:
I20260928 08:53:36.912930 30342 leader_election.cc:288] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:53:36.913007 30347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:53:36.913247 30347 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [term 1 LEADER]: Becoming Leader. State: Replica: 0b528077fc484f4fae48815cbca7e762, State: Running, Role: LEADER
I20260928 08:53:36.913257 30342 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:53:36.913415 30347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } }
I20260928 08:53:36.914070 30348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b528077fc484f4fae48815cbca7e762" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } }
I20260928 08:53:36.914176 30348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:36.914098 30349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b528077fc484f4fae48815cbca7e762. Latest consensus state: current_term: 1 leader_uuid: "0b528077fc484f4fae48815cbca7e762" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } }
I20260928 08:53:36.914443 30349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:36.914780 30356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:53:36.915484 30356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:53:36.917131 30356 catalog_manager.cc:1347] Generated new cluster ID: 671a81755e8346ab80e440c4bc391553
I20260928 08:53:36.917191 30356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:53:36.947485 30356 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:53:36.948251 30356 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:53:36.958105 30356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762: Generated new TSK 0
I20260928 08:53:36.958349 30356 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.167.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.167.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.167.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.193.keytab.
Entry for principal oryx/127.23.167.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.193.keytab.
W20260928 08:53:37.648777 24223 mini_kdc.cc:262] Time spent creating service keytab for oryx/127.23.167.193: real 0.688s user 0.000s sys 0.003s
I20260928 08:53:37.649107 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:0
--local_ip_for_outbound_sockets=127.23.167.193
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:34791
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.193.keytab
--principal=oryx/127.23.167.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:36571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:37.727324 30368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:37.727550 30368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:37.727573 30368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:37.728732 30368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:37.728772 30368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:37.728797 30368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:53:37.730275 30368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:36571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.193.keytab
--principal=oryx/127.23.167.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:34791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:37.730507 30368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:37.730722 30368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:37.735617 30373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:37.735697 30368 server_base.cc:1034] running on GCE node
W20260928 08:53:37.735625 30376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:37.735625 30374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:37.736069 30368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:37.736307 30368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:37.737438 30368 hybrid_clock.cc:648] HybridClock initialized: now 1790585617737424 us; error 27 us; skew 500 ppm
Sep 28 08:53:37 dist-test-slave-pbjp krb5kdc[30261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585617, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:37.743212 30368 init.cc:377] Logged in from keytab as oryx/127.23.167.193@KRBTEST.COM (short username oryx)
I20260928 08:53:37.743710 30368 webserver.cc:460] Webserver started at http://127.23.167.193:46847/ using document root <none> and password file <none>
I20260928 08:53:37.743942 30368 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:37.743988 30368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:37.744191 30368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:37.745386 30368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-0/data/instance:
uuid: "513c83afb10d4d09b2f13357edabd418"
format_stamp: "Formatted at 2026-09-28 08:53:37 on dist-test-slave-pbjp"
server_key: "a5202d0d03ee8f4467bf2ed39f6523d8"
server_key_iv: "d004d9dbfa5a5906a25d52b3b059aea6"
server_key_version: "encryptionkey@0"
I20260928 08:53:37.745730 30368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance:
uuid: "513c83afb10d4d09b2f13357edabd418"
format_stamp: "Formatted at 2026-09-28 08:53:37 on dist-test-slave-pbjp"
server_key: "a5202d0d03ee8f4467bf2ed39f6523d8"
server_key_iv: "d004d9dbfa5a5906a25d52b3b059aea6"
server_key_version: "encryptionkey@0"
I20260928 08:53:37.747080 30368 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:53:37.747848 30383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:37.748034 30368 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:53:37.748123 30368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "513c83afb10d4d09b2f13357edabd418"
format_stamp: "Formatted at 2026-09-28 08:53:37 on dist-test-slave-pbjp"
server_key: "a5202d0d03ee8f4467bf2ed39f6523d8"
server_key_iv: "d004d9dbfa5a5906a25d52b3b059aea6"
server_key_version: "encryptionkey@0"
I20260928 08:53:37.748183 30368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:37.764966 30368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:37.767021 30368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:37.767138 30368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:37.767369 30368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:53:37.767649 30368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:37.767680 30368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:37.767704 30368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:37.767715 30368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:37.773129 30368 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:38813
I20260928 08:53:37.773186 30496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:38813 every 8 connection(s)
I20260928 08:53:37.773500 30368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
I20260928 08:53:37.773770 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 30368
I20260928 08:53:37.773876 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance
I20260928 08:53:37.774075 24223 external_mini_cluster.cc:1428] Setting key 8f0a072729c4a56e4d9504f9b54f09f2
Sep 28 08:53:37 dist-test-slave-pbjp krb5kdc[30261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585617, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.23.167.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.167.194@KRBTEST.COM" created.
I20260928 08:53:37.783507 30499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:37.774235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:36425 (local address 127.23.167.254:34791)
0928 08:53:37.774348 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:37.774350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:37.774550 (+ 200us) server_negotiation.cc:407] Connection header received
0928 08:53:37.775111 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:37.775113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:37.775126 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:37.775191 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:37.776340 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:37.776838 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:37.777951 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:37.778155 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:37.778209 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:37.780185 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:37.780186 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:37.780187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:37.780194 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:37.781860 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:37.782357 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:37.782358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:37.782359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:37.782431 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:37.782769 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:37.782771 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:37.782771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:37.782925 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:37.782994 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:37.783321 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 08:53:37.783411 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":53,"threads_started":1}
I20260928 08:53:37.783685 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:37.774232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.193:36425)
0928 08:53:37.774421 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:37.774466 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:37.775012 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:37.775277 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:37.775282 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:37.775913 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:37.776186 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:37.776193 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:37.776921 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:37.776922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:37.777873 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:37.777876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:37.778221 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:37.778222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:37.778358 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:37.778810 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:37.778818 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:37.780102 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:37.781944 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:37.781951 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:37.781953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:37.782259 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:37.782497 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:37.782498 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:37.782498 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:37.782635 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:37.782986 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:37.782994 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:37.783155 (+ 161us) client_negotiation.cc:769] Sending connection context
0928 08:53:37.783288 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 08:53:37.783452 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":101,"threads_started":1}
I20260928 08:53:37.784294 30497 heartbeater.cc:347] Connected to a master server at 127.23.167.254:34791
I20260928 08:53:37.784389 30497 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:37.784577 30497 heartbeater.cc:511] Master 127.23.167.254:34791 requested a full tablet report, sending...
I20260928 08:53:37.785172 30306 ts_manager.cc:194] Registered new tserver with Master: 513c83afb10d4d09b2f13357edabd418 (127.23.167.193:38813)
I20260928 08:53:37.785770 30306 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.167.193@KRBTEST.COM'} at 127.23.167.193:36425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.167.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.194.keytab.
Entry for principal oryx/127.23.167.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.194.keytab.
I20260928 08:53:37.797447 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:0
--local_ip_for_outbound_sockets=127.23.167.194
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:34791
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.194.keytab
--principal=oryx/127.23.167.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:36571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:37.876948 30503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:37.877166 30503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:37.877197 30503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:37.878298 30503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:37.878345 30503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:37.878378 30503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:53:37.879776 30503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:36571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.194.keytab
--principal=oryx/127.23.167.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:34791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:37.880005 30503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:37.880198 30503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:37.884850 30511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:37.884858 30508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:37.884945 30503 server_base.cc:1034] running on GCE node
W20260928 08:53:37.884858 30509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:37.885301 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:37.885522 30503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:37.886675 30503 hybrid_clock.cc:648] HybridClock initialized: now 1790585617886662 us; error 29 us; skew 500 ppm
Sep 28 08:53:37 dist-test-slave-pbjp krb5kdc[30261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585617, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:37.889107 30503 init.cc:377] Logged in from keytab as oryx/127.23.167.194@KRBTEST.COM (short username oryx)
I20260928 08:53:37.889567 30503 webserver.cc:460] Webserver started at http://127.23.167.194:43653/ using document root <none> and password file <none>
I20260928 08:53:37.889770 30503 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:37.889816 30503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:37.889940 30503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:37.890923 30503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-1/data/instance:
uuid: "e75b3311a1d5482ea942df5c55632674"
format_stamp: "Formatted at 2026-09-28 08:53:37 on dist-test-slave-pbjp"
server_key: "c85afdf2dff6f1acb0a9afbec932e557"
server_key_iv: "f9074d790d4af838e625b52b726cc8bc"
server_key_version: "encryptionkey@0"
I20260928 08:53:37.891239 30503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance:
uuid: "e75b3311a1d5482ea942df5c55632674"
format_stamp: "Formatted at 2026-09-28 08:53:37 on dist-test-slave-pbjp"
server_key: "c85afdf2dff6f1acb0a9afbec932e557"
server_key_iv: "f9074d790d4af838e625b52b726cc8bc"
server_key_version: "encryptionkey@0"
I20260928 08:53:37.892482 30503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:53:37.893139 30518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:37.893307 30503 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:37.893391 30503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "e75b3311a1d5482ea942df5c55632674"
format_stamp: "Formatted at 2026-09-28 08:53:37 on dist-test-slave-pbjp"
server_key: "c85afdf2dff6f1acb0a9afbec932e557"
server_key_iv: "f9074d790d4af838e625b52b726cc8bc"
server_key_version: "encryptionkey@0"
I20260928 08:53:37.893463 30503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:37.911921 30503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:37.914130 30503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:37.914268 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:37.914494 30503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:53:37.914803 30503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:37.914844 30503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:37.914875 30503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:37.914902 30503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:37.920336 30503 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:38385
I20260928 08:53:37.920396 30631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:38385 every 8 connection(s)
I20260928 08:53:37.920696 30503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
I20260928 08:53:37.921686 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 30503
I20260928 08:53:37.921785 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance
I20260928 08:53:37.922009 24223 external_mini_cluster.cc:1428] Setting key e270d7d8f5dcdb869a838594e318cf7d
Sep 28 08:53:37 dist-test-slave-pbjp krb5kdc[30261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585617, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.23.167.195@KRBTEST.COM; defaulting to no policy
I20260928 08:53:37.930927 30499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:37.921438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:39933 (local address 127.23.167.254:34791)
0928 08:53:37.921486 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:37.921487 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:37.921746 (+ 259us) server_negotiation.cc:407] Connection header received
0928 08:53:37.922448 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:37.922449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:37.922460 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:37.922506 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:37.923800 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:37.924478 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:37.925562 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:37.925788 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:37.925852 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:37.928246 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:37.928247 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:37.928247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:37.928255 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:37.929512 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:37.930009 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:37.930010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:37.930010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:37.930058 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:37.930322 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:37.930322 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:37.930322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:37.930422 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:37.930467 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:37.930777 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 08:53:37.930858 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 08:53:37.931164 30634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:37.921436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.194:39933)
0928 08:53:37.921587 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:37.921638 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:37.922338 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:37.922595 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:37.922600 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:37.923404 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:37.923688 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:37.923694 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:37.924575 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:37.924576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:37.925486 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:37.925489 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:37.925888 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:37.925889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:37.926033 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:37.926534 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:37.926543 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:37.928153 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:37.929618 (+ 1465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:37.929626 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:37.929628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:37.929941 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:37.930120 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:37.930120 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:37.930121 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:37.930260 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:37.930474 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:37.930479 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:37.930642 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 08:53:37.930762 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 08:53:37.930937 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20260928 08:53:37.931622 30632 heartbeater.cc:347] Connected to a master server at 127.23.167.254:34791
I20260928 08:53:37.931722 30632 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:37.931901 30632 heartbeater.cc:511] Master 127.23.167.254:34791 requested a full tablet report, sending...
I20260928 08:53:37.932271 30306 ts_manager.cc:194] Registered new tserver with Master: e75b3311a1d5482ea942df5c55632674 (127.23.167.194:38385)
I20260928 08:53:37.932642 30306 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.167.194@KRBTEST.COM'} at 127.23.167.194:39933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.167.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.167.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.195.keytab.
Entry for principal oryx/127.23.167.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.195.keytab.
I20260928 08:53:37.948282 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:0
--local_ip_for_outbound_sockets=127.23.167.195
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:34791
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.195.keytab
--principal=oryx/127.23.167.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:36571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:38.027781 30637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:38.027971 30637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:38.027992 30637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:38.029121 30637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:38.029158 30637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:38.029184 30637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:53:38.030571 30637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:36571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.195.keytab
--principal=oryx/127.23.167.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:34791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:38.030776 30637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:38.030982 30637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 08:53:38.035754 30637 server_base.cc:1034] running on GCE node
W20260928 08:53:38.035730 30642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:38.035744 30643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:38.035737 30645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:38.036115 30637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:38.036332 30637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:38.037495 30637 hybrid_clock.cc:648] HybridClock initialized: now 1790585618037471 us; error 38 us; skew 500 ppm
Sep 28 08:53:38 dist-test-slave-pbjp krb5kdc[30261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585618, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:38.039928 30637 init.cc:377] Logged in from keytab as oryx/127.23.167.195@KRBTEST.COM (short username oryx)
I20260928 08:53:38.040307 30637 webserver.cc:460] Webserver started at http://127.23.167.195:44005/ using document root <none> and password file <none>
I20260928 08:53:38.040490 30637 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:38.040529 30637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:38.040614 30637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:38.041606 30637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-2/data/instance:
uuid: "f5265171404243129b6726d8707066be"
format_stamp: "Formatted at 2026-09-28 08:53:38 on dist-test-slave-pbjp"
server_key: "7c23fcd00ab2ec86ff681c7cb281ed0d"
server_key_iv: "db7dab64a08f601237317d763b80ba99"
server_key_version: "encryptionkey@0"
I20260928 08:53:38.041889 30637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance:
uuid: "f5265171404243129b6726d8707066be"
format_stamp: "Formatted at 2026-09-28 08:53:38 on dist-test-slave-pbjp"
server_key: "7c23fcd00ab2ec86ff681c7cb281ed0d"
server_key_iv: "db7dab64a08f601237317d763b80ba99"
server_key_version: "encryptionkey@0"
I20260928 08:53:38.043026 30637 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:53:38.043658 30652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:38.043797 30637 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:53:38.043884 30637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "f5265171404243129b6726d8707066be"
format_stamp: "Formatted at 2026-09-28 08:53:38 on dist-test-slave-pbjp"
server_key: "7c23fcd00ab2ec86ff681c7cb281ed0d"
server_key_iv: "db7dab64a08f601237317d763b80ba99"
server_key_version: "encryptionkey@0"
I20260928 08:53:38.043952 30637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:38.078877 30637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:38.081130 30637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:38.081250 30637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:38.081486 30637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:53:38.081799 30637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:38.081841 30637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:38.081863 30637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:38.081887 30637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:38.087144 30637 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:41903
I20260928 08:53:38.087200 30765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:41903 every 8 connection(s)
I20260928 08:53:38.087486 30637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
I20260928 08:53:38.093253 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 30637
Sep 28 08:53:38 dist-test-slave-pbjp krb5kdc[30261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585618, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM
I20260928 08:53:38.093335 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance
I20260928 08:53:38.093484 24223 external_mini_cluster.cc:1428] Setting key 5609d6fa2098c6acd542365698abc727
I20260928 08:53:38.096311 30499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.088118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:55143 (local address 127.23.167.254:34791)
0928 08:53:38.088199 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:38.088201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:38.088360 (+ 159us) server_negotiation.cc:407] Connection header received
0928 08:53:38.088853 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:38.088855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:38.088865 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:38.088918 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:38.089917 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.090427 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.091414 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.091638 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.091692 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.093839 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:38.093841 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:38.093841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:38.093848 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:38.095072 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.095426 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.095428 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.095428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.095487 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.095695 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.095695 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.095696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.095808 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:38.095884 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:38.096137 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 08:53:38.096220 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 08:53:38.096493 30768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.088084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.195:55143)
0928 08:53:38.088242 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:38.088279 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:38.088752 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:38.088984 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:38.088987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:38.089589 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:38.089810 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.089815 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.090501 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.090502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.091297 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.091299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.091700 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.091700 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.091803 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.092276 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:38.092284 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:38.093747 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:38.095144 (+ 1397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.095151 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.095153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.095362 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.095541 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.095541 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.095542 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.095641 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.095881 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:38.095889 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:38.096021 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 08:53:38.096108 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 08:53:38.096285 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
I20260928 08:53:38.096902 30766 heartbeater.cc:347] Connected to a master server at 127.23.167.254:34791
I20260928 08:53:38.096984 30766 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:38.097195 30766 heartbeater.cc:511] Master 127.23.167.254:34791 requested a full tablet report, sending...
I20260928 08:53:38.097535 30306 ts_manager.cc:194] Registered new tserver with Master: f5265171404243129b6726d8707066be (127.23.167.195:41903)
I20260928 08:53:38.097885 30306 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.167.195@KRBTEST.COM'} at 127.23.167.195:55143
I20260928 08:53:38.105131 24223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.167.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.167.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.167.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.253.keytab.
Entry for principal oryx/127.23.167.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.253.keytab.
I20260928 08:53:38.128124 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.253:44651
--webserver_interface=127.23.167.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.253.keytab
--principal=oryx/127.23.167.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:36571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:34791,127.23.167.253:44651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:38.208171 30771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:38.208408 30771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:38.208432 30771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:38.209700 30771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:53:38.209745 30771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:38.209759 30771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:38.209771 30771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:53:38.209782 30771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:53:38.211283 30771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:36571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:34791,127.23.167.253:44651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.253.keytab
--principal=oryx/127.23.167.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.253:44651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.167.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
W20260928 08:53:38.211395 30771 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.23.167.254:34791,127.23.167.253:44651'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 08:53:38.211512 30771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:38.211701 30771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:38.216646 30776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:38.216624 30777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:38.216656 30771 server_base.cc:1034] running on GCE node
W20260928 08:53:38.216625 30779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:38.217134 30771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:38.217453 30771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:38.218618 30771 hybrid_clock.cc:648] HybridClock initialized: now 1790585618218604 us; error 30 us; skew 500 ppm
Sep 28 08:53:38 dist-test-slave-pbjp krb5kdc[30261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585618, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:38.221132 30771 init.cc:377] Logged in from keytab as oryx/127.23.167.253@KRBTEST.COM (short username oryx)
I20260928 08:53:38.221637 30771 webserver.cc:460] Webserver started at http://127.23.167.253:34007/ using document root <none> and password file <none>
I20260928 08:53:38.221870 30771 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:38.221920 30771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:38.222050 30771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:38.223176 30771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-1/data/instance:
uuid: "940646d9191e48d3903cb7037d8ed33c"
format_stamp: "Formatted at 2026-09-28 08:53:38 on dist-test-slave-pbjp"
server_key: "ff49635c5461f3d05e1789e2793bdbc7"
server_key_iv: "ae943182fec98079a2e9236ac2f34f85"
server_key_version: "encryptionkey@0"
I20260928 08:53:38.223512 30771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-1/wal/instance:
uuid: "940646d9191e48d3903cb7037d8ed33c"
format_stamp: "Formatted at 2026-09-28 08:53:38 on dist-test-slave-pbjp"
server_key: "ff49635c5461f3d05e1789e2793bdbc7"
server_key_iv: "ae943182fec98079a2e9236ac2f34f85"
server_key_version: "encryptionkey@0"
I20260928 08:53:38.224767 30771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 08:53:38.225536 30786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:38.225687 30771 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:38.225767 30771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-1/wal
uuid: "940646d9191e48d3903cb7037d8ed33c"
format_stamp: "Formatted at 2026-09-28 08:53:38 on dist-test-slave-pbjp"
server_key: "ff49635c5461f3d05e1789e2793bdbc7"
server_key_iv: "ae943182fec98079a2e9236ac2f34f85"
server_key_version: "encryptionkey@0"
I20260928 08:53:38.225849 30771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:38.258713 30771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:38.260860 30771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:38.260994 30771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:38.264643 30771 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.253:44651
I20260928 08:53:38.264689 30838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.253:44651 every 8 connection(s)
I20260928 08:53:38.265027 30771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-1/data/info.pb
I20260928 08:53:38.266003 30839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:38.267546 30839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:53:38.272893 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 30771
Sep 28 08:53:38 dist-test-slave-pbjp krb5kdc[30261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585618, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.253@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM
I20260928 08:53:38.276921 30499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.268256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47680 (local address 127.23.167.254:34791)
0928 08:53:38.268321 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:38.268323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:38.268524 (+ 201us) server_negotiation.cc:407] Connection header received
0928 08:53:38.268967 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:38.268968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:38.268978 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:38.269039 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:38.269635 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.270252 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.271252 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.271479 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.271535 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.273810 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:38.273812 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:38.273813 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:38.273821 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:38.275481 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.275904 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.275905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.275905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.275960 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.276210 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.276210 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.276211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.276389 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:38.276442 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:38.276752 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 08:53:38.276829 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 08:53:38.277019 30841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.268233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.0.0.1:47680)
0928 08:53:38.268408 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:38.268450 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:38.268888 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:38.269126 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:38.269129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:38.269368 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:38.269563 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.269568 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.270342 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.270343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.271169 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.271172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.271544 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.271544 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.271645 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.272110 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:38.272119 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:38.273728 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:38.275573 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.275579 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.275581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.275830 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.276014 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.276015 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.276015 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.276149 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.276447 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:38.276452 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:38.276607 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 08:53:38.276712 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 08:53:38.276843 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":88,"threads_started":1}
I20260928 08:53:38.277678 30839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.167.253" port: 44651 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:53:38 dist-test-slave-pbjp krb5kdc[30261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.167.253@KRBTEST.COM
Sep 28 08:53:38 dist-test-slave-pbjp krb5kdc[30261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585618, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.253@KRBTEST.COM for oryx/127.23.167.253@KRBTEST.COM
I20260928 08:53:38.282585 30843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.273479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49294 (local address 127.23.167.253:44651)
0928 08:53:38.273634 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:38.273639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:38.273670 (+ 31us) server_negotiation.cc:407] Connection header received
0928 08:53:38.273694 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:38.273696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:38.273715 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:38.273783 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:38.274701 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.275711 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.276625 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.277049 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.277136 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.279558 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:38.279565 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:38.279565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:38.279572 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:38.281328 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.281764 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.281767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.281768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.281835 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.282045 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.282046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.282046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.282181 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:38.282317 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:38.282409 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 08:53:38.282481 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260928 08:53:38.284548 30841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.277918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:44651 (local address 127.0.0.1:49302)
0928 08:53:38.277956 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:38.277966 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:38.278001 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:38.278223 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:38.278223 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:38.278336 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:38.278379 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.278380 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.279054 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.279055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.279582 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.279584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.279989 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.279990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.280042 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.280389 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:38.280394 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:38.281764 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:38.283746 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.283747 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.283747 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.283894 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.284052 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.284052 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.284052 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.284094 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.284307 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:38.284307 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:38.284378 (+ 71us) client_negotiation.cc:769] Sending connection context
0928 08:53:38.284422 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 08:53:38.284489 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 08:53:38.284657 30844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.277947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49302 (local address 127.23.167.253:44651)
0928 08:53:38.278104 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:38.278106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:38.278113 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:53:38.278132 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:38.278132 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:38.278142 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:38.278221 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:38.278440 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.278983 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.279647 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.279916 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.279980 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.281853 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:38.281854 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:38.281854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:38.281860 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:38.283681 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.283948 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.283949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.283949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.283993 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.284150 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.284151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.284151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.284258 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:38.284318 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:38.284448 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 08:53:38.284533 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":137,"thread_start_us":100,"threads_started":1}
I20260928 08:53:38.285823 30839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c: Bootstrap starting.
I20260928 08:53:38.286391 30839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:38.286630 30839 log.cc:824] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:38.287256 30839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c: No bootstrap required, opened a new log
I20260928 08:53:38.288568 30839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 44651 } }
I20260928 08:53:38.288686 30839 raft_consensus.cc:374] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:38.288717 30839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 940646d9191e48d3903cb7037d8ed33c, State: Initialized, Role: FOLLOWER
I20260928 08:53:38.288829 30839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 44651 } }
I20260928 08:53:38.289136 30839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:53:38.289099 30846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 44651 } } }
I20260928 08:53:38.289192 30846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [sys.catalog]: This master's current role is: FOLLOWER
W20260928 08:53:38.290632 30857 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:53:38.290686 30857 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:53:38.292172 30771 master_runner.cc:428] Detected that this master 940646d9191e48d3903cb7037d8ed33c is joining an existing cluster
I20260928 08:53:38.292209 30771 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.167.253:44651
I20260928 08:53:38.298918 30844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.293057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49318 (local address 127.23.167.253:44651)
0928 08:53:38.293112 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:38.293114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:38.293440 (+ 326us) server_negotiation.cc:407] Connection header received
0928 08:53:38.293463 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:38.293463 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:38.293472 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:38.293516 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:38.293855 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.294333 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.295051 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.295257 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.295323 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.296191 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:38.296192 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:38.296192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:38.296198 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:38.297874 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.298230 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.298232 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.298232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.298287 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.298464 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.298465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.298465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.298574 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:38.298628 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:38.298776 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 08:53:38.298847 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 08:53:38.299340 30864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.293156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:44651 (local address 127.0.0.1:49318)
0928 08:53:38.293390 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:38.293396 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:38.293431 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:38.293519 (+ 88us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:38.293520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:38.293735 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:38.293792 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.293794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.294401 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.294402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.294979 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.294981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.295379 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.295379 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.295421 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.295803 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:38.295808 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:38.296093 (+ 285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:38.297968 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.297969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.297970 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.298162 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.298356 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.298357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.298357 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.298417 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.298629 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:38.298630 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:38.298677 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 08:53:38.299014 (+ 337us) client_negotiation.cc:240] Negotiation successful
0928 08:53:38.299098 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":1,"thread_start_us":152,"threads_started":1}
I20260928 08:53:38.299553 30499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.293059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47690 (local address 127.23.167.254:34791)
0928 08:53:38.293124 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:38.293126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:38.293318 (+ 192us) server_negotiation.cc:407] Connection header received
0928 08:53:38.293368 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:38.293369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:38.293381 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:38.293436 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:38.293889 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.294691 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.295755 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.296044 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.296106 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.297026 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:38.297027 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:38.297027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:38.297034 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:38.298353 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.298683 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.298684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.298685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.298743 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.298922 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.298922 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.298923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.299042 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:38.299096 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:38.299397 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 08:53:38.299467 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 08:53:38.299652 30863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.293024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.0.0.1:47690)
0928 08:53:38.293236 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:38.293247 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:38.293314 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:38.293482 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:38.293483 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:38.293732 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:38.293803 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.293805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.294781 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.294782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.295666 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.295669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.296119 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.296120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.296202 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.296572 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:38.296576 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:38.296942 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:38.298417 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.298418 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.298418 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.298604 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.298807 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.298808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.298809 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.298862 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.299119 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:38.299120 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:38.299158 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 08:53:38.299214 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 08:53:38.299328 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":131,"threads_started":1}
Sep 28 08:53:38 dist-test-slave-pbjp krb5kdc[30261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585616, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.254@KRBTEST.COM for oryx/127.23.167.253@KRBTEST.COM
I20260928 08:53:38.308962 30865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.301677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:44651 (local address 127.0.0.1:49322)
0928 08:53:38.301787 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:38.301832 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:38.301900 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:38.302023 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:38.302026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:38.302283 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:38.302343 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.302345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.303227 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.303227 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.303834 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.303835 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.304264 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.304265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.304320 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.304794 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:38.304799 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:38.306398 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:38.307982 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.307991 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.307993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.308202 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.308388 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.308389 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.308389 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.308439 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.308674 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:38.308679 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:38.308726 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 08:53:38.308789 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 08:53:38.308886 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":60,"threads_started":1}
I20260928 08:53:38.309003 30844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.301732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49322 (local address 127.23.167.253:44651)
0928 08:53:38.301769 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:38.301771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:38.301893 (+ 122us) server_negotiation.cc:407] Connection header received
0928 08:53:38.301954 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:38.301954 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:38.301964 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:38.302019 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:38.302435 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.303152 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.303906 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.304202 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.304276 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.306488 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:38.306490 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:38.306490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:38.306498 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:38.307877 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.308266 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.308267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.308268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.308332 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.308488 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.308488 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.308489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.308620 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:38.308682 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:38.308828 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 08:53:38.308909 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 08:53:38.309602 30306 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.167.253:44651: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "940646d9191e48d3903cb7037d8ed33c"
member_type: NON_VOTER
last_known_addr {
host: "127.23.167.253"
port: 44651
}
attrs {
promote: true
}
}
dest_uuid: "0b528077fc484f4fae48815cbca7e762"
cas_config_opid_index: -1
I20260928 08:53:38.309902 30306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: NON_VOTER last_known_addr { host: "127.23.167.253" port: 44651 } attrs { promote: true } }
I20260928 08:53:38.310444 30868 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 940646d9191e48d3903cb7037d8ed33c (127.23.167.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: NON_VOTER last_known_addr { host: "127.23.167.253" port: 44651 } attrs { promote: true } } }
I20260928 08:53:38.310753 30813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:38.311523 30868 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.167.253:44651
I20260928 08:53:38.311594 30813 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 1 FOLLOWER]: Refusing update from remote peer 0b528077fc484f4fae48815cbca7e762: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 08:53:38.311621 30868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0b528077fc484f4fae48815cbca7e762" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: NON_VOTER last_known_addr { host: "127.23.167.253" port: 44651 } attrs { promote: true } } }
I20260928 08:53:38.311674 30868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:38.311781 30771 master.cc:537] Master@127.23.167.253:44651 shutting down...
I20260928 08:53:38.312207 30868 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [LEADER]: Connected to new peer: Peer: permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: NON_VOTER last_known_addr { host: "127.23.167.253" port: 44651 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 08:53:38.313129 30293 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 08:53:38.313751 30293 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 -> Peer 940646d9191e48d3903cb7037d8ed33c (127.23.167.253:44651): Couldn't send request to peer 940646d9191e48d3903cb7037d8ed33c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 08:53:38.314708 30771 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:53:38.314795 30771 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:53:38.314846 30771 tablet_replica.cc:330] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c: stopping tablet replica
I20260928 08:53:38.327661 30771 master.cc:552] Master@127.23.167.253:44651 shutdown complete.
I20260928 08:53:38.327711 30771 master_runner.cc:305] Clearing existing system tablet
I20260928 08:53:38.327826 30771 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:38.327870 30771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:38.328460 30771 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:53:38.329051 30872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:38.329182 30771 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:38.329234 30771 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 08:53:38.329257 30771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-1/wal
uuid: "940646d9191e48d3903cb7037d8ed33c"
format_stamp: "Formatted at 2026-09-28 08:53:38 on dist-test-slave-pbjp"
server_key: "ff49635c5461f3d05e1789e2793bdbc7"
server_key_iv: "ae943182fec98079a2e9236ac2f34f85"
server_key_version: "encryptionkey@0"
I20260928 08:53:38.329903 30771 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:53:38.331283 30771 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 08:53:38.331342 30771 log.cc:1196] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 08:53:38.331521 30771 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c: Deleting consensus metadata
I20260928 08:53:38.331625 30771 master_runner.cc:315] Copying system tablet from 127.23.167.254:34791
I20260928 08:53:38.332194 30771 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c: tablet copy: Beginning tablet copy session from remote peer at address 127.23.167.254:34791
I20260928 08:53:38.338248 30499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.332595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47702 (local address 127.23.167.254:34791)
0928 08:53:38.332648 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:38.332650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:38.332748 (+ 98us) server_negotiation.cc:407] Connection header received
0928 08:53:38.332820 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:38.332821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:38.332831 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:38.332874 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:38.333339 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.333968 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.334689 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.334889 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.334943 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.335825 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:38.335827 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:38.335827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:38.335835 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:38.337135 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.337482 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.337483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.337484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.337548 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.337711 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.337711 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.337712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.337823 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:38.337894 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:38.338080 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 08:53:38.338154 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 08:53:38.338258 30878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.332536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.0.0.1:47702)
0928 08:53:38.332670 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:38.332681 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:38.332756 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:38.332878 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:38.332878 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:38.333196 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:38.333256 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.333259 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.334045 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.334046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.334610 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.334612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.334948 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.334949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.335019 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.335442 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:38.335447 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:38.335739 (+ 292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:38.337214 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.337215 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.337216 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.337426 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.337600 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.337601 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.337601 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.337660 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.337921 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:38.337922 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:38.337969 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 08:53:38.338044 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 08:53:38.338151 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
I20260928 08:53:38.338799 30326 tablet_copy_service.cc:140] P 0b528077fc484f4fae48815cbca7e762: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 940646d9191e48d3903cb7037d8ed33c ({username='oryx', principal='oryx/127.23.167.253@KRBTEST.COM'} at 127.0.0.1:47702)
I20260928 08:53:38.338914 30326 tablet_copy_service.cc:161] P 0b528077fc484f4fae48815cbca7e762: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 940646d9191e48d3903cb7037d8ed33c at {username='oryx', principal='oryx/127.23.167.253@KRBTEST.COM'} at 127.0.0.1:47702: session id = 940646d9191e48d3903cb7037d8ed33c-00000000000000000000000000000000
I20260928 08:53:38.339802 30326 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 08:53:38.340607 30771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 08:53:38.341697 30771 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c: tablet copy: Starting download of 0 data blocks...
I20260928 08:53:38.341831 30771 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c: tablet copy: Starting download of 1 WAL segments...
I20260928 08:53:38.342623 30771 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 08:53:38.343382 30326 tablet_copy_service.cc:342] P 0b528077fc484f4fae48815cbca7e762: Request end of tablet copy session 940646d9191e48d3903cb7037d8ed33c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.167.253@KRBTEST.COM'} at 127.0.0.1:47702
I20260928 08:53:38.343442 30326 tablet_copy_service.cc:433] P 0b528077fc484f4fae48815cbca7e762: ending tablet copy session 940646d9191e48d3903cb7037d8ed33c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 940646d9191e48d3903cb7037d8ed33c
I20260928 08:53:38.345821 30771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:38.346851 30879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:38.346907 30880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:38.347048 30882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:38.347476 30771 server_base.cc:1034] running on GCE node
I20260928 08:53:38.347569 30771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:38.347698 30771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:38.348812 30771 hybrid_clock.cc:648] HybridClock initialized: now 1790585618348794 us; error 33 us; skew 500 ppm
Sep 28 08:53:38 dist-test-slave-pbjp krb5kdc[30261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585618, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:38.350198 30771 init.cc:377] Logged in from keytab as oryx/127.23.167.253@KRBTEST.COM (short username oryx)
I20260928 08:53:38.350445 30771 webserver.cc:460] Webserver started at http://127.23.167.253:39485/ using document root <none> and password file <none>
I20260928 08:53:38.350558 30771 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:38.350593 30771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:38.351188 30771 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:38.351706 30889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:38.351823 30771 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:38.351864 30771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-1/wal
uuid: "940646d9191e48d3903cb7037d8ed33c"
format_stamp: "Formatted at 2026-09-28 08:53:38 on dist-test-slave-pbjp"
server_key: "ff49635c5461f3d05e1789e2793bdbc7"
server_key_iv: "ae943182fec98079a2e9236ac2f34f85"
server_key_version: "encryptionkey@0"
I20260928 08:53:38.351922 30771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:38.361434 30771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:38.362867 30771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:38.365341 30771 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.253:44651
I20260928 08:53:38.365728 30771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-1/data/info.pb
I20260928 08:53:38.365762 30941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.253:44651 every 8 connection(s)
I20260928 08:53:38.366623 30942 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:53:38.367192 30942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c: Bootstrap starting.
I20260928 08:53:38.370123 30942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:53:38.370361 30942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c: Bootstrap complete.
I20260928 08:53:38.370565 30942 raft_consensus.cc:348] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: NON_VOTER last_known_addr { host: "127.23.167.253" port: 44651 } attrs { promote: true } }
I20260928 08:53:38.370652 30942 raft_consensus.cc:729] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 940646d9191e48d3903cb7037d8ed33c, State: Initialized, Role: LEARNER
I20260928 08:53:38.370690 30942 consensus_queue.cc:260] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: NON_VOTER last_known_addr { host: "127.23.167.253" port: 44651 } attrs { promote: true } }
I20260928 08:53:38.370885 30942 sys_catalog.cc:564] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:53:38.370859 30945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: NON_VOTER last_known_addr { host: "127.23.167.253" port: 44651 } attrs { promote: true } } }
I20260928 08:53:38.370949 30945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [sys.catalog]: This master's current role is: LEARNER
I20260928 08:53:38.372367 30956 catalog_manager.cc:1259] Loaded cluster ID: 671a81755e8346ab80e440c4bc391553
I20260928 08:53:38.372435 30956 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c: loading cluster ID for follower catalog manager: success
I20260928 08:53:38.373400 30956 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c: acquiring CA information for follower catalog manager: success
I20260928 08:53:38.373662 30956 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 08:53:38.389971 30957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.384152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49340 (local address 127.23.167.253:44651)
0928 08:53:38.384267 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:38.384268 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:38.384275 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:53:38.384295 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:38.384296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:38.384305 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:38.384365 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:38.384888 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.385589 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.386270 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.386467 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.386521 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.387663 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:38.387665 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:38.387665 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:38.387672 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:38.388932 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.389266 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.389267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.389268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.389322 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.389481 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.389483 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.389483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.389584 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:38.389690 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:38.389779 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 08:53:38.389855 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
WARNING: no policy specified for oryx/127.23.167.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.167.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.167.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.252.keytab.
Entry for principal oryx/127.23.167.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.252.keytab.
I20260928 08:53:38.415308 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.252:40727
--webserver_interface=127.23.167.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.252.keytab
--principal=oryx/127.23.167.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:36571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:34791,127.23.167.253:44651,127.23.167.252:40727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:53:38.496482 30960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:38.496687 30960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:38.496707 30960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:38.497927 30960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:53:38.497970 30960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:38.497984 30960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:38.497995 30960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:53:38.498005 30960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:53:38.499518 30960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:36571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:34791,127.23.167.253:44651,127.23.167.252:40727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.252.keytab
--principal=oryx/127.23.167.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.252:40727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.167.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:38.499745 30960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:38.499965 30960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 08:53:38.504808 30960 server_base.cc:1034] running on GCE node
W20260928 08:53:38.504788 30968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:38.504779 30965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:38.504788 30966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:38.505290 30960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:38.505590 30960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:38.506737 30960 hybrid_clock.cc:648] HybridClock initialized: now 1790585618506723 us; error 29 us; skew 500 ppm
Sep 28 08:53:38 dist-test-slave-pbjp krb5kdc[30261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585618, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:38.509198 30960 init.cc:377] Logged in from keytab as oryx/127.23.167.252@KRBTEST.COM (short username oryx)
I20260928 08:53:38.509641 30960 webserver.cc:460] Webserver started at http://127.23.167.252:46635/ using document root <none> and password file <none>
I20260928 08:53:38.509858 30960 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:38.509902 30960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:38.510021 30960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:53:38.511088 30960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-2/data/instance:
uuid: "ce49edb20df84405acc0d56445850999"
format_stamp: "Formatted at 2026-09-28 08:53:38 on dist-test-slave-pbjp"
server_key: "2ef7f59e22c498cb1985d47082e8898a"
server_key_iv: "08fcccb4e8cd49bfc6df9183543f3182"
server_key_version: "encryptionkey@0"
I20260928 08:53:38.511409 30960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-2/wal/instance:
uuid: "ce49edb20df84405acc0d56445850999"
format_stamp: "Formatted at 2026-09-28 08:53:38 on dist-test-slave-pbjp"
server_key: "2ef7f59e22c498cb1985d47082e8898a"
server_key_iv: "08fcccb4e8cd49bfc6df9183543f3182"
server_key_version: "encryptionkey@0"
I20260928 08:53:38.512616 30960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 08:53:38.513410 30975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:38.513566 30960 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:53:38.513660 30960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-2/wal
uuid: "ce49edb20df84405acc0d56445850999"
format_stamp: "Formatted at 2026-09-28 08:53:38 on dist-test-slave-pbjp"
server_key: "2ef7f59e22c498cb1985d47082e8898a"
server_key_iv: "08fcccb4e8cd49bfc6df9183543f3182"
server_key_version: "encryptionkey@0"
I20260928 08:53:38.513736 30960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:38.526911 30960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:38.528685 30960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:38.528831 30960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:38.533041 30960 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.252:40727
I20260928 08:53:38.533092 31027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.252:40727 every 8 connection(s)
I20260928 08:53:38.533442 30960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-2/data/info.pb
I20260928 08:53:38.534399 31028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:38.535910 31028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:53:38.540238 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 30960
Sep 28 08:53:38 dist-test-slave-pbjp krb5kdc[30261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585618, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.252@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM
I20260928 08:53:38.545176 30499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.536656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47712 (local address 127.23.167.254:34791)
0928 08:53:38.536730 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:38.536733 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:38.536874 (+ 141us) server_negotiation.cc:407] Connection header received
0928 08:53:38.537439 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:38.537441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:38.537454 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:38.537504 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:38.538145 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.538644 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.539684 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.539881 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.539936 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.542247 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:38.542249 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:38.542249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:38.542256 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:38.543747 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.544174 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.544175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.544176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.544248 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.544530 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.544531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.544531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.544649 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:38.544705 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:38.544968 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 08:53:38.545049 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 08:53:38.545310 31030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.536559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.0.0.1:47712)
0928 08:53:38.536742 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:38.536787 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:38.537325 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:38.537578 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:38.537581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:38.537839 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:38.538051 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.538056 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.538728 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.538729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.539597 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.539599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.539965 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.539967 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.540107 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.540614 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:38.540622 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:38.542140 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:38.543804 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.543809 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.543811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.544087 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.544322 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.544322 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.544322 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.544461 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.544720 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:38.544726 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:38.544845 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 08:53:38.544931 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 08:53:38.545097 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":94,"threads_started":1}
I20260928 08:53:38.545851 31028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.167.253" port: 44651 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:53:38 dist-test-slave-pbjp krb5kdc[30261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.167.252@KRBTEST.COM
I20260928 08:53:38.549288 31031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.540743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50666 (local address 127.23.167.252:40727)
0928 08:53:38.540903 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:38.540908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:38.540927 (+ 19us) server_negotiation.cc:407] Connection header received
0928 08:53:38.540960 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:38.540961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:38.540983 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:38.541045 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:38.541622 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.542553 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.543596 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.544045 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.544124 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.546484 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:38.546493 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:38.546493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:38.546499 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:38.548100 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.548525 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.548528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.548528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.548575 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.548760 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.548760 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.548761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.548920 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:38.549054 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:38.549120 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 08:53:38.549187 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
Sep 28 08:53:38 dist-test-slave-pbjp krb5kdc[30261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585618, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.252@KRBTEST.COM for oryx/127.23.167.253@KRBTEST.COM
I20260928 08:53:38.551955 31030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.546092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:44651 (local address 127.0.0.1:49348)
0928 08:53:38.546173 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:38.546181 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:38.546214 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:38.546341 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:38.546342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:38.546462 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:38.546513 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.546514 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.547133 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.547134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.547893 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.547896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.548301 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.548302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.548360 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.548666 (+ 306us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:38.548670 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:38.549925 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:38.551147 (+ 1222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.551148 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.551148 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.551318 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.551510 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.551510 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.551511 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.551557 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.551749 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:38.551749 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:38.551789 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 08:53:38.551834 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 08:53:38.551896 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 08:53:38.551980 30957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.546168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49348 (local address 127.23.167.253:44651)
0928 08:53:38.546213 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:38.546215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:38.546222 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:53:38.546267 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:38.546268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:38.546280 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:38.546331 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:38.546580 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.547048 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.547990 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.548219 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.548306 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.550014 (+ 1708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:38.550015 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:38.550016 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:38.550022 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:38.551070 (+ 1048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.551419 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.551419 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.551420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.551468 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.551590 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.551590 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.551590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.551679 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:38.551749 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:38.551852 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 08:53:38.551919 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 08:53:38.552341 31028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.167.252" port: 40727 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:53:38 dist-test-slave-pbjp krb5kdc[30261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585618, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.252@KRBTEST.COM for oryx/127.23.167.252@KRBTEST.COM
I20260928 08:53:38.557603 31030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.552539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:40727 (local address 127.0.0.1:50678)
0928 08:53:38.552577 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:38.552585 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:38.552616 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:38.552701 (+ 85us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:38.552702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:38.552836 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:38.552864 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.552865 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.553500 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.553500 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.553948 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.553949 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.554246 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.554247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.554278 (+ 31us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.554583 (+ 305us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:38.554587 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:38.555613 (+ 1026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:38.556777 (+ 1164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.556777 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.556778 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.556926 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.557105 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.557105 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.557106 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.557157 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.557372 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:38.557372 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:38.557403 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 08:53:38.557455 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 08:53:38.557523 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 08:53:38.557651 31031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.552581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50678 (local address 127.23.167.252:40727)
0928 08:53:38.552626 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:38.552628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:38.552634 (+ 6us) server_negotiation.cc:407] Connection header received
0928 08:53:38.552649 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:38.552649 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:38.552657 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:38.552699 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:38.552915 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.553446 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.554005 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.554200 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.554246 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.555684 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:38.555685 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:38.555685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:38.555691 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:38.556692 (+ 1001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.556989 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.556989 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.556990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.557030 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.557198 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.557198 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.557198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.557327 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:38.557376 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:38.557472 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 08:53:38.557564 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":29}
I20260928 08:53:38.558897 31028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999: Bootstrap starting.
I20260928 08:53:38.559713 31028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:38.559967 31028 log.cc:824] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:38.560632 31028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999: No bootstrap required, opened a new log
I20260928 08:53:38.562166 31028 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 44651 } } peers { permanent_uuid: "ce49edb20df84405acc0d56445850999" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 40727 } }
I20260928 08:53:38.562299 31028 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:38.562319 31028 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce49edb20df84405acc0d56445850999, State: Initialized, Role: FOLLOWER
I20260928 08:53:38.562435 31028 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 44651 } } peers { permanent_uuid: "ce49edb20df84405acc0d56445850999" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 40727 } }
I20260928 08:53:38.562741 31028 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:53:38.562711 31033 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 44651 } } peers { permanent_uuid: "ce49edb20df84405acc0d56445850999" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 40727 } } }
I20260928 08:53:38.562788 31033 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 08:53:38.564263 31044 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:53:38.564316 31044 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:53:38.566407 30960 master_runner.cc:428] Detected that this master ce49edb20df84405acc0d56445850999 is joining an existing cluster
I20260928 08:53:38.566455 30960 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.167.252:40727
I20260928 08:53:38.573990 31031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.567806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50680 (local address 127.23.167.252:40727)
0928 08:53:38.567845 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:38.567847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:38.568116 (+ 269us) server_negotiation.cc:407] Connection header received
0928 08:53:38.568183 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:38.568184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:38.568193 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:38.568237 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:38.568686 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.569428 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.570511 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.570778 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.570811 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.571761 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:38.571762 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:38.571763 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:38.571768 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:38.572994 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.573309 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.573310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.573310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.573357 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.573506 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.573507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.573507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.573604 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:38.573657 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:38.573799 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 08:53:38.573875 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 08:53:38.574057 31050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.567750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:40727 (local address 127.0.0.1:50680)
0928 08:53:38.568068 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:38.568076 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:38.568124 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:38.568271 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:38.568271 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:38.568526 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:38.568613 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.568615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.569534 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.569535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.570422 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.570424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.570886 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.570887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.570938 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.571371 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:38.571375 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:38.571687 (+ 312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:38.573108 (+ 1421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.573109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.573109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.573252 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.573407 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.573408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.573408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.573458 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.573652 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:38.573653 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:38.573715 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 08:53:38.573783 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 08:53:38.573926 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":63,"thread_start_us":63,"threads_started":1}
I20260928 08:53:38.592814 30957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.567523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49350 (local address 127.23.167.253:44651)
0928 08:53:38.567565 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:38.567567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:38.568265 (+ 698us) server_negotiation.cc:407] Connection header received
0928 08:53:38.568350 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:38.568351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:38.568362 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:38.568389 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:38.568791 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.569530 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.570513 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.570824 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.570859 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.590033 (+ 19174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:38.590035 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:38.590035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:38.590043 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:38.591629 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.592085 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.592086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.592086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.592144 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.592300 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.592301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.592302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.592431 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:38.592493 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:38.592644 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 08:53:38.592726 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 08:53:38.592860 30499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.567603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47728 (local address 127.23.167.254:34791)
0928 08:53:38.567646 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:38.567647 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:38.568106 (+ 459us) server_negotiation.cc:407] Connection header received
0928 08:53:38.568160 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:38.568161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:38.568173 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:38.568233 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:38.568732 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.569538 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.570600 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.570919 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.570961 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.590128 (+ 19167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:38.590129 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:38.590130 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:38.590138 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:38.591730 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.592115 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.592117 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.592117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.592178 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.592365 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.592366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.592366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.592491 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:38.592544 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:38.592699 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 08:53:38.592771 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 08:53:38.592861 31051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.567911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:44651 (local address 127.0.0.1:49350)
0928 08:53:38.568214 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:38.568222 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:38.568286 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:38.568461 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:38.568462 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:38.568668 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:38.568718 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.568719 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.569609 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.569609 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.570426 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.570428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.570940 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.570941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.571011 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.589552 (+ 18541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:38.589558 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:38.589926 (+ 368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:38.591719 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.591720 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.591721 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.592009 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.592203 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.592203 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.592204 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.592254 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.592494 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:38.592495 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:38.592553 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 08:53:38.592621 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 08:53:38.592724 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":2,"thread_start_us":72,"threads_started":1}
I20260928 08:53:38.593288 31052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.567527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.0.0.1:47728)
0928 08:53:38.568028 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:38.568039 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:38.568104 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:38.568282 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:38.568283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:38.568567 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:38.568638 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.568641 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.569630 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.569631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.570502 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.570504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.571054 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.571055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.571116 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.589610 (+ 18494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:38.589616 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:38.590036 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:38.591788 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.591789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.591789 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.592022 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.592249 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.592250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.592250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.592308 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.592550 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:38.592551 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:38.592593 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 08:53:38.593043 (+ 450us) client_negotiation.cc:240] Negotiation successful
0928 08:53:38.593176 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":89,"threads_started":1}
Sep 28 08:53:38 dist-test-slave-pbjp krb5kdc[30261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585616, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.254@KRBTEST.COM for oryx/127.23.167.252@KRBTEST.COM
I20260928 08:53:38.600528 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 30368
I20260928 08:53:38.601675 31031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.595275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50692 (local address 127.23.167.252:40727)
0928 08:53:38.595343 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:38.595345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:38.595366 (+ 21us) server_negotiation.cc:407] Connection header received
0928 08:53:38.595434 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:38.595435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:38.595445 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:38.595485 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:38.595727 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.596188 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.596849 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.597022 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.597097 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.598875 (+ 1778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:38.598877 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:38.598877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:38.598884 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:38.600193 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.600539 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.600540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.600540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.601014 (+ 474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.601222 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.601223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.601223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.601323 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:38.601377 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:38.601516 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 08:53:38.601586 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 08:53:38.601681 30865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.595228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:40727 (local address 127.0.0.1:50692)
0928 08:53:38.595320 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:38.595332 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:38.595380 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:38.595494 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:38.595494 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:38.595630 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:38.595669 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.595671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.596250 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.596251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.596777 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.596779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.597095 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.597096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.597175 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.597486 (+ 311us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:38.597490 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:38.598785 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:38.600270 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.600271 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.600271 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.600471 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.601102 (+ 631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.601103 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.601104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.601167 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.601381 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:38.601382 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:38.601427 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 08:53:38.601489 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 08:53:38.601587 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 08:53:38.606062 30306 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.167.252:40727: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ce49edb20df84405acc0d56445850999"
member_type: NON_VOTER
last_known_addr {
host: "127.23.167.252"
port: 40727
}
attrs {
promote: true
}
}
dest_uuid: "0b528077fc484f4fae48815cbca7e762"
cas_config_opid_index: 5
I20260928 08:53:38.606319 30306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: NON_VOTER last_known_addr { host: "127.23.167.253" port: 44651 } attrs { promote: true } } peers { permanent_uuid: "ce49edb20df84405acc0d56445850999" member_type: NON_VOTER last_known_addr { host: "127.23.167.252" port: 40727 } attrs { promote: true } }
I20260928 08:53:38.606680 30867 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ce49edb20df84405acc0d56445850999 (127.23.167.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: NON_VOTER last_known_addr { host: "127.23.167.253" port: 44651 } attrs { promote: true } } peers { permanent_uuid: "ce49edb20df84405acc0d56445850999" member_type: NON_VOTER last_known_addr { host: "127.23.167.252" port: 40727 } attrs { promote: true } } }
I20260928 08:53:38.607556 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:38813
--local_ip_for_outbound_sockets=127.23.167.193
--tserver_master_addrs=127.23.167.254:34791
--webserver_port=46847
--webserver_interface=127.23.167.193
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.193.keytab
--principal=oryx/127.23.167.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:36571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.167.254:34791,127.23.167.253:44651
--tserver_master_addrs=127.23.167.254:34791,127.23.167.253:44651,127.23.167.252:40727 with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:53:38.607698 30867 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.167.252:40727
I20260928 08:53:38.607769 30867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0b528077fc484f4fae48815cbca7e762" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: NON_VOTER last_known_addr { host: "127.23.167.253" port: 44651 } attrs { promote: true } } peers { permanent_uuid: "ce49edb20df84405acc0d56445850999" member_type: NON_VOTER last_known_addr { host: "127.23.167.252" port: 40727 } attrs { promote: true } } }
I20260928 08:53:38.607826 30867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:38.608090 30960 master.cc:537] Master@127.23.167.252:40727 shutting down...
I20260928 08:53:38.608114 31002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:38.610328 31002 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999 [term 1 FOLLOWER]: Refusing update from remote peer 0b528077fc484f4fae48815cbca7e762: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 08:53:38.610610 30868 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce49edb20df84405acc0d56445850999" member_type: NON_VOTER last_known_addr { host: "127.23.167.252" port: 40727 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 08:53:38.611605 30291 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 -> Peer ce49edb20df84405acc0d56445850999 (127.23.167.252:40727): Couldn't send request to peer ce49edb20df84405acc0d56445850999. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 08:53:38.612191 30957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.606887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49356 (local address 127.23.167.253:44651)
0928 08:53:38.606938 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:38.606940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:38.606947 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:53:38.606970 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:38.606971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:38.606994 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:38.607060 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:53:38.607199 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.607888 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.611067 (+ 3179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.611746 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.611811 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.611910 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:38.612091 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 08:53:38.612131 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 08:53:38.612336 30865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.606778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:44651 (local address 127.0.0.1:49356)
0928 08:53:38.606824 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:38.606842 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:38.606881 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:38.607083 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:38.607083 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:38.607085 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:53:38.607122 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.607124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.609827 (+ 2703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.609828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.610968 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.610971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.611942 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.611943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.612006 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.612032 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 08:53:38.612085 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 08:53:38.612125 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 08:53:38.612596 30916 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 1 LEARNER]: Refusing update from remote peer 0b528077fc484f4fae48815cbca7e762: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 08:53:38.612771 30868 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [LEADER]: Connected to new peer: Peer: permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: NON_VOTER last_known_addr { host: "127.23.167.253" port: 44651 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 08:53:38.613200 30960 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:53:38.613286 30960 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:53:38.613346 30960 tablet_replica.cc:330] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999: stopping tablet replica
I20260928 08:53:38.613556 30916 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ce49edb20df84405acc0d56445850999 (127.23.167.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: NON_VOTER last_known_addr { host: "127.23.167.253" port: 44651 } attrs { promote: true } } peers { permanent_uuid: "ce49edb20df84405acc0d56445850999" member_type: NON_VOTER last_known_addr { host: "127.23.167.252" port: 40727 } attrs { promote: true } } }
I20260928 08:53:38.614604 31055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b528077fc484f4fae48815cbca7e762. Latest consensus state: current_term: 1 leader_uuid: "0b528077fc484f4fae48815cbca7e762" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: NON_VOTER last_known_addr { host: "127.23.167.253" port: 44651 } attrs { promote: true } } peers { permanent_uuid: "ce49edb20df84405acc0d56445850999" member_type: NON_VOTER last_known_addr { host: "127.23.167.252" port: 40727 } attrs { promote: true } } }
I20260928 08:53:38.614687 31055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [sys.catalog]: This master's current role is: LEARNER
I20260928 08:53:38.614717 31055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b528077fc484f4fae48815cbca7e762" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: NON_VOTER last_known_addr { host: "127.23.167.253" port: 44651 } attrs { promote: true } } peers { permanent_uuid: "ce49edb20df84405acc0d56445850999" member_type: NON_VOTER last_known_addr { host: "127.23.167.252" port: 40727 } attrs { promote: true } } }
I20260928 08:53:38.614750 31055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [sys.catalog]: This master's current role is: LEARNER
I20260928 08:53:38.615461 30868 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762: attempting to promote NON_VOTER 940646d9191e48d3903cb7037d8ed33c to VOTER
I20260928 08:53:38.615651 30868 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 44651 } attrs { promote: false } } peers { permanent_uuid: "ce49edb20df84405acc0d56445850999" member_type: NON_VOTER last_known_addr { host: "127.23.167.252" port: 40727 } attrs { promote: true } }
I20260928 08:53:38.616292 30916 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 1 LEARNER]: Refusing update from remote peer 0b528077fc484f4fae48815cbca7e762: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 08:53:38.616519 30865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.616354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:40727 (local address 127.0.0.1:50708)
0928 08:53:38.616450 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:38.616491 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.252:40727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 08:53:38.616577 30867 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [LEADER]: Connected to new peer: Peer: permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 44651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 08:53:38.616935 30865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.616860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:40727 (local address 127.0.0.1:50714)
0928 08:53:38.616890 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:38.616910 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.252:40727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260928 08:53:38.616987 30291 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 -> Peer ce49edb20df84405acc0d56445850999 (127.23.167.252:40727): Couldn't send request to peer ce49edb20df84405acc0d56445850999. Status: Network error: Client connection negotiation failed: client connection to 127.23.167.252:40727: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 08:53:38.617338 30867 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 940646d9191e48d3903cb7037d8ed33c (127.23.167.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 44651 } attrs { promote: false } } peers { permanent_uuid: "ce49edb20df84405acc0d56445850999" member_type: NON_VOTER last_known_addr { host: "127.23.167.252" port: 40727 } attrs { promote: true } } }
I20260928 08:53:38.618006 30867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0b528077fc484f4fae48815cbca7e762" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 44651 } attrs { promote: false } } peers { permanent_uuid: "ce49edb20df84405acc0d56445850999" member_type: NON_VOTER last_known_addr { host: "127.23.167.252" port: 40727 } attrs { promote: true } } }
I20260928 08:53:38.618083 30867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:38.618144 31056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b528077fc484f4fae48815cbca7e762" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 44651 } attrs { promote: false } } peers { permanent_uuid: "ce49edb20df84405acc0d56445850999" member_type: NON_VOTER last_known_addr { host: "127.23.167.252" port: 40727 } attrs { promote: true } } }
I20260928 08:53:38.618252 31056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762 [sys.catalog]: This master's current role is: LEADER
I20260928 08:53:38.618304 30916 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 940646d9191e48d3903cb7037d8ed33c (127.23.167.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 44651 } attrs { promote: false } } peers { permanent_uuid: "ce49edb20df84405acc0d56445850999" member_type: NON_VOTER last_known_addr { host: "127.23.167.252" port: 40727 } attrs { promote: true } } }
I20260928 08:53:38.619104 31055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b528077fc484f4fae48815cbca7e762" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 44651 } attrs { promote: false } } peers { permanent_uuid: "ce49edb20df84405acc0d56445850999" member_type: NON_VOTER last_known_addr { host: "127.23.167.252" port: 40727 } attrs { promote: true } } }
I20260928 08:53:38.619184 31055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:53:38.627079 30960 master.cc:552] Master@127.23.167.252:40727 shutdown complete.
I20260928 08:53:38.627127 30960 master_runner.cc:305] Clearing existing system tablet
I20260928 08:53:38.627249 30960 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:38.627290 30960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:38.627861 30960 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:53:38.628314 31058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:38.628448 30960 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:38.628494 30960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:38.628521 30960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-2/wal
uuid: "ce49edb20df84405acc0d56445850999"
format_stamp: "Formatted at 2026-09-28 08:53:38 on dist-test-slave-pbjp"
server_key: "2ef7f59e22c498cb1985d47082e8898a"
server_key_iv: "08fcccb4e8cd49bfc6df9183543f3182"
server_key_version: "encryptionkey@0"
I20260928 08:53:38.629300 30960 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:53:38.630659 30960 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 08:53:38.630731 30960 log.cc:1196] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 08:53:38.630926 30960 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999: Deleting consensus metadata
I20260928 08:53:38.631048 30960 master_runner.cc:315] Copying system tablet from 127.23.167.254:34791
I20260928 08:53:38.631594 30960 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999: tablet copy: Beginning tablet copy session from remote peer at address 127.23.167.254:34791
I20260928 08:53:38.637768 31064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.631948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.0.0.1:47738)
0928 08:53:38.632078 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:38.632088 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:38.632147 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:38.632280 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:38.632281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:38.632452 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:38.632505 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.632507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.633268 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.633270 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.633990 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.633994 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.634382 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.634383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.634449 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.634891 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:38.634896 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:38.635193 (+ 297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:38.636648 (+ 1455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.636649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.636649 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.636820 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.636990 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.636991 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.636992 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.637046 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.637317 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:38.637317 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:38.637366 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 08:53:38.637537 (+ 171us) client_negotiation.cc:240] Negotiation successful
0928 08:53:38.637654 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":73,"threads_started":1}
I20260928 08:53:38.637764 30499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.632010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47738 (local address 127.23.167.254:34791)
0928 08:53:38.632056 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:38.632059 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:38.632132 (+ 73us) server_negotiation.cc:407] Connection header received
0928 08:53:38.632208 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:38.632209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:38.632219 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:38.632262 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:38.632578 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.633186 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.634097 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.634315 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.634374 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.635281 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:38.635282 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:38.635283 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:38.635291 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:38.636569 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.636881 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.636883 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.636883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.636938 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.637130 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.637131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.637132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.637259 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:38.637317 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:38.637567 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 08:53:38.637667 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 08:53:38.638303 30326 tablet_copy_service.cc:140] P 0b528077fc484f4fae48815cbca7e762: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ce49edb20df84405acc0d56445850999 ({username='oryx', principal='oryx/127.23.167.252@KRBTEST.COM'} at 127.0.0.1:47738)
I20260928 08:53:38.638363 30326 tablet_copy_service.cc:161] P 0b528077fc484f4fae48815cbca7e762: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ce49edb20df84405acc0d56445850999 at {username='oryx', principal='oryx/127.23.167.252@KRBTEST.COM'} at 127.0.0.1:47738: session id = ce49edb20df84405acc0d56445850999-00000000000000000000000000000000
I20260928 08:53:38.638808 30326 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 08:53:38.639251 30960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 08:53:38.640231 30960 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999: tablet copy: Starting download of 0 data blocks...
I20260928 08:53:38.640373 30960 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999: tablet copy: Starting download of 1 WAL segments...
I20260928 08:53:38.641145 30960 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 08:53:38.641948 30326 tablet_copy_service.cc:342] P 0b528077fc484f4fae48815cbca7e762: Request end of tablet copy session ce49edb20df84405acc0d56445850999-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.167.252@KRBTEST.COM'} at 127.0.0.1:47738
I20260928 08:53:38.642000 30326 tablet_copy_service.cc:433] P 0b528077fc484f4fae48815cbca7e762: ending tablet copy session ce49edb20df84405acc0d56445850999-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ce49edb20df84405acc0d56445850999
I20260928 08:53:38.644114 30960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:38.645289 31068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:38.645300 31065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:38.645670 31066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:38.645903 30960 server_base.cc:1034] running on GCE node
I20260928 08:53:38.645994 30960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:38.646135 30960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:38.647244 30960 hybrid_clock.cc:648] HybridClock initialized: now 1790585618647230 us; error 32 us; skew 500 ppm
Sep 28 08:53:38 dist-test-slave-pbjp krb5kdc[30261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585618, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:38.648644 30960 init.cc:377] Logged in from keytab as oryx/127.23.167.252@KRBTEST.COM (short username oryx)
I20260928 08:53:38.648903 30960 webserver.cc:460] Webserver started at http://127.23.167.252:37745/ using document root <none> and password file <none>
I20260928 08:53:38.649014 30960 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:38.649047 30960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:38.649649 30960 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:38.650130 31075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:38.650249 30960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:38.650290 30960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-2/wal
uuid: "ce49edb20df84405acc0d56445850999"
format_stamp: "Formatted at 2026-09-28 08:53:38 on dist-test-slave-pbjp"
server_key: "2ef7f59e22c498cb1985d47082e8898a"
server_key_iv: "08fcccb4e8cd49bfc6df9183543f3182"
server_key_version: "encryptionkey@0"
I20260928 08:53:38.650352 30960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:38.657341 30960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:38.658787 30960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:38.661514 30960 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.252:40727
I20260928 08:53:38.661778 31127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.252:40727 every 8 connection(s)
I20260928 08:53:38.661877 30960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/master-2/data/info.pb
I20260928 08:53:38.662590 31128 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:53:38.662989 31128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999: Bootstrap starting.
I20260928 08:53:38.665140 31128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:53:38.665366 31128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999: Bootstrap complete.
I20260928 08:53:38.665558 31128 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 44651 } attrs { promote: false } } peers { permanent_uuid: "ce49edb20df84405acc0d56445850999" member_type: NON_VOTER last_known_addr { host: "127.23.167.252" port: 40727 } attrs { promote: true } }
I20260928 08:53:38.665630 31128 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ce49edb20df84405acc0d56445850999, State: Initialized, Role: LEARNER
I20260928 08:53:38.665665 31128 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 44651 } attrs { promote: false } } peers { permanent_uuid: "ce49edb20df84405acc0d56445850999" member_type: NON_VOTER last_known_addr { host: "127.23.167.252" port: 40727 } attrs { promote: true } }
I20260928 08:53:38.665814 31128 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:53:38.665812 31131 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 44651 } attrs { promote: false } } peers { permanent_uuid: "ce49edb20df84405acc0d56445850999" member_type: NON_VOTER last_known_addr { host: "127.23.167.252" port: 40727 } attrs { promote: true } } }
I20260928 08:53:38.666059 31131 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999 [sys.catalog]: This master's current role is: LEARNER
I20260928 08:53:38.667927 31142 catalog_manager.cc:1259] Loaded cluster ID: 671a81755e8346ab80e440c4bc391553
I20260928 08:53:38.667963 31142 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999: loading cluster ID for follower catalog manager: success
I20260928 08:53:38.668655 31142 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999: acquiring CA information for follower catalog manager: success
I20260928 08:53:38.668906 31142 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ce49edb20df84405acc0d56445850999: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 08:53:38.717969 31053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:38.718180 31053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:38.718202 31053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:38.719406 31053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:38.719449 31053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:38.719475 31053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:53:38.720954 31053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:36571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.193.keytab
--principal=oryx/127.23.167.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:38813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=46847
--tserver_master_addrs=127.23.167.254:34791,127.23.167.253:44651,127.23.167.252:40727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:38.721215 31053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:38.721454 31053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:38.726481 31148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:38.726444 31150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:38.726564 31053 server_base.cc:1034] running on GCE node
W20260928 08:53:38.726444 31147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:38.726842 31053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:38.727092 31053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:38.728245 31053 hybrid_clock.cc:648] HybridClock initialized: now 1790585618728232 us; error 29 us; skew 500 ppm
Sep 28 08:53:38 dist-test-slave-pbjp krb5kdc[30261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585618, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:38.730937 31053 init.cc:377] Logged in from keytab as oryx/127.23.167.193@KRBTEST.COM (short username oryx)
I20260928 08:53:38.731410 31053 webserver.cc:460] Webserver started at http://127.23.167.193:46847/ using document root <none> and password file <none>
I20260928 08:53:38.731635 31053 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:38.731701 31053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:38.733105 31053 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:53:38.733721 31157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:38.733894 31053 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:38.733978 31053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "513c83afb10d4d09b2f13357edabd418"
format_stamp: "Formatted at 2026-09-28 08:53:37 on dist-test-slave-pbjp"
server_key: "a5202d0d03ee8f4467bf2ed39f6523d8"
server_key_iv: "d004d9dbfa5a5906a25d52b3b059aea6"
server_key_version: "encryptionkey@0"
I20260928 08:53:38.734370 31053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:38.764446 31053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:38.766896 31053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:38.767061 31053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:38.767310 31053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:53:38.767616 31053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:38.767649 31053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:38.767679 31053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:38.767701 31053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:38.773689 31053 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:38813
I20260928 08:53:38.773748 31270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:38813 every 8 connection(s)
I20260928 08:53:38.774080 31053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
Sep 28 08:53:38 dist-test-slave-pbjp krb5kdc[30261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585618, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.253@KRBTEST.COM
Sep 28 08:53:38 dist-test-slave-pbjp krb5kdc[30261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585618, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM
Sep 28 08:53:38 dist-test-slave-pbjp krb5kdc[30261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585618, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.252@KRBTEST.COM
I20260928 08:53:38.783888 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 31053
I20260928 08:53:38.784045 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 30503
I20260928 08:53:38.785126 30957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.775417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:51329 (local address 127.23.167.253:44651)
0928 08:53:38.775471 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:38.775472 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:38.775482 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:53:38.776098 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:38.776099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:38.776111 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:38.776150 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:38.776960 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.777747 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.778935 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.779215 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.779285 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.781521 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:38.781522 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:38.781522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:38.781529 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:38.783108 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.783566 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.783567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.783568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.783627 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.783871 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.783873 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.783873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.784023 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:38.784084 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:38.784935 (+ 851us) server_negotiation.cc:299] Negotiation successful
0928 08:53:38.785015 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 08:53:38.785693 31278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.775296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:44651 (local address 127.23.167.193:51329)
0928 08:53:38.775422 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:38.775431 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:38.776014 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:38.776224 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:38.776228 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:38.776649 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:38.776872 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.776877 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.777836 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.777837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.778847 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.778850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.779293 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.779294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.779359 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.779810 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:38.779820 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:38.781437 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:38.783195 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.783202 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.783205 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.783495 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.783694 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.783695 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.783695 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.783813 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.784075 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:38.784082 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:38.784772 (+ 690us) client_negotiation.cc:769] Sending connection context
0928 08:53:38.785257 (+ 485us) client_negotiation.cc:240] Negotiation successful
0928 08:53:38.785447 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":91,"thread_start_us":66,"threads_started":1}
I20260928 08:53:38.786772 31275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.774739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:37091 (local address 127.23.167.252:40727)
0928 08:53:38.774860 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:38.774862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:38.775034 (+ 172us) server_negotiation.cc:407] Connection header received
0928 08:53:38.776122 (+ 1088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:38.776124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:38.776137 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:38.776179 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:38.776978 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.777687 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.778956 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.779167 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.779225 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.782566 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:38.782567 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:38.782567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:38.782575 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:38.785466 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.786063 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.786064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.786064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.786121 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.786284 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.786284 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.786285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.786398 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:38.786433 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:38.786602 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 08:53:38.786677 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":48,"threads_started":1}
I20260928 08:53:38.786899 31273 heartbeater.cc:347] Connected to a master server at 127.23.167.253:44651
I20260928 08:53:38.786999 31273 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:38.787185 31273 heartbeater.cc:511] Master 127.23.167.253:44651 requested a full tablet report, sending...
I20260928 08:53:38.787354 31276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.774742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:40727 (local address 127.23.167.193:37091)
0928 08:53:38.774905 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:38.774942 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:38.776002 (+ 1060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:38.776240 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:38.776241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:38.776661 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:38.776873 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.776879 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.777784 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.777784 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.778861 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.778862 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.779296 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.779297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.779362 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.780010 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:38.780014 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:38.782368 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:38.785544 (+ 3176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.785545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.785546 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.785985 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.786182 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.786182 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.786183 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.786231 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.786470 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:38.786471 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:38.786508 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 08:53:38.787111 (+ 603us) client_negotiation.cc:240] Negotiation successful
0928 08:53:38.787209 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":113,"thread_start_us":85,"threads_started":1}
I20260928 08:53:38.788130 30906 ts_manager.cc:194] Registered new tserver with Master: 513c83afb10d4d09b2f13357edabd418 (127.23.167.193:38813)
I20260928 08:53:38.788154 31271 heartbeater.cc:347] Connected to a master server at 127.23.167.252:40727
I20260928 08:53:38.788193 31271 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:38.788251 31271 heartbeater.cc:511] Master 127.23.167.252:40727 requested a full tablet report, sending...
I20260928 08:53:38.788581 31092 ts_manager.cc:194] Registered new tserver with Master: 513c83afb10d4d09b2f13357edabd418 (127.23.167.193:38813)
I20260928 08:53:38.789732 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:38385
--local_ip_for_outbound_sockets=127.23.167.194
--tserver_master_addrs=127.23.167.254:34791
--webserver_port=43653
--webserver_interface=127.23.167.194
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.194.keytab
--principal=oryx/127.23.167.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:36571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.167.254:34791,127.23.167.253:44651
--tserver_master_addrs=127.23.167.254:34791,127.23.167.253:44651,127.23.167.252:40727 with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:53:38.790187 31277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.775018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.193:48947)
0928 08:53:38.775154 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:38.775163 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:38.776007 (+ 844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:38.776240 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:38.776241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:38.776652 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:38.776873 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.776879 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.777858 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.777859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.778680 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.778683 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.779168 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.779168 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.779329 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.780093 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:38.780097 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:38.781889 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:38.783482 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.783483 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.783483 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.783653 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.789640 (+ 5987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.789640 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.789641 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.789695 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.789956 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:38.789957 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:38.789994 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 08:53:38.790042 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 08:53:38.790118 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":47,"thread_start_us":70,"threads_started":1}
I20260928 08:53:38.790238 30499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.775094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:48947 (local address 127.23.167.254:34791)
0928 08:53:38.775139 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:38.775141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:38.775218 (+ 77us) server_negotiation.cc:407] Connection header received
0928 08:53:38.776123 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:38.776124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:38.776136 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:38.776170 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:38.776993 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.777777 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.778762 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.779090 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.779156 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.782000 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:38.782001 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:38.782002 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:38.782009 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:38.783396 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.789502 (+ 6106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.789503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.789504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.789575 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.789754 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.789755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.789755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.789897 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:38.789954 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:38.790077 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 08:53:38.790155 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 08:53:38.790539 31272 heartbeater.cc:347] Connected to a master server at 127.23.167.254:34791
I20260928 08:53:38.790584 31272 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:38.790640 31272 heartbeater.cc:511] Master 127.23.167.254:34791 requested a full tablet report, sending...
I20260928 08:53:38.790881 30306 ts_manager.cc:194] Re-registered known tserver with Master: 513c83afb10d4d09b2f13357edabd418 (127.23.167.193:38813)
I20260928 08:53:38.791366 30306 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.167.193@KRBTEST.COM'} at 127.23.167.193:48947
W20260928 08:53:38.871485 31279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:38.871712 31279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:38.871735 31279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:38.872869 31279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:38.872913 31279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:38.872938 31279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:53:38.874384 31279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:36571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.194.keytab
--principal=oryx/127.23.167.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:38385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=43653
--tserver_master_addrs=127.23.167.254:34791,127.23.167.253:44651,127.23.167.252:40727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:38.874574 31279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:38.874749 31279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:38.879307 31287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:38.879318 31284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:38.879323 31285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:38.879514 31279 server_base.cc:1034] running on GCE node
I20260928 08:53:38.879750 31279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:38.879959 31279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:38.881121 31279 hybrid_clock.cc:648] HybridClock initialized: now 1790585618881101 us; error 28 us; skew 500 ppm
Sep 28 08:53:38 dist-test-slave-pbjp krb5kdc[30261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585618, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:38.883466 31279 init.cc:377] Logged in from keytab as oryx/127.23.167.194@KRBTEST.COM (short username oryx)
I20260928 08:53:38.883858 31279 webserver.cc:460] Webserver started at http://127.23.167.194:43653/ using document root <none> and password file <none>
I20260928 08:53:38.884076 31279 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:38.884136 31279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:38.885444 31279 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:53:38.886040 31294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:38.886184 31279 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:53:38.886262 31279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "e75b3311a1d5482ea942df5c55632674"
format_stamp: "Formatted at 2026-09-28 08:53:37 on dist-test-slave-pbjp"
server_key: "c85afdf2dff6f1acb0a9afbec932e557"
server_key_iv: "f9074d790d4af838e625b52b726cc8bc"
server_key_version: "encryptionkey@0"
I20260928 08:53:38.886590 31279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:38.895560 31279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:38.897257 31279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:38.897382 31279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:38.897570 31279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:53:38.897845 31279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:38.897881 31279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:38.897908 31279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:38.897925 31279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:38.903167 31279 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:38385
I20260928 08:53:38.903223 31407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:38385 every 8 connection(s)
I20260928 08:53:38.903522 31279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
I20260928 08:53:38.904423 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 31279
I20260928 08:53:38.904578 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 30637
I20260928 08:53:38.909987 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:41903
--local_ip_for_outbound_sockets=127.23.167.195
--tserver_master_addrs=127.23.167.254:34791
--webserver_port=44005
--webserver_interface=127.23.167.195
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.195.keytab
--principal=oryx/127.23.167.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:36571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.167.254:34791,127.23.167.253:44651
--tserver_master_addrs=127.23.167.254:34791,127.23.167.253:44651,127.23.167.252:40727 with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:53:38 dist-test-slave-pbjp krb5kdc[30261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585618, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM
Sep 28 08:53:38 dist-test-slave-pbjp krb5kdc[30261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585618, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.252@KRBTEST.COM
Sep 28 08:53:38 dist-test-slave-pbjp krb5kdc[30261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585618, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.253@KRBTEST.COM
I20260928 08:53:38.916004 31275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.904169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:47423 (local address 127.23.167.252:40727)
0928 08:53:38.904555 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:38.904557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:38.904568 (+ 11us) server_negotiation.cc:407] Connection header received
0928 08:53:38.906369 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:38.906371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:38.906385 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:38.906461 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:38.907629 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.908390 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.909976 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.910253 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.910294 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.913017 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:38.913019 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:38.913019 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:38.913027 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:38.914533 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.914971 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.914972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.914973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.915023 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.915278 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.915279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.915279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.915426 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:38.915465 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:38.915823 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 08:53:38.915908 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357}
I20260928 08:53:38.916004 30957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.904167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:37445 (local address 127.23.167.253:44651)
0928 08:53:38.904237 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:38.904239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:38.904494 (+ 255us) server_negotiation.cc:407] Connection header received
0928 08:53:38.906368 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:38.906370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:38.906383 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:38.906414 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:38.907626 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.908399 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.911494 (+ 3095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.911754 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.911823 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.913384 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:38.913385 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:38.913386 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:38.913393 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:38.914842 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.915222 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.915223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.915223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.915276 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.915446 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.915447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.915447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.915565 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:38.915609 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:38.915832 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 08:53:38.915907 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 08:53:38.916577 31413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.904175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:44651 (local address 127.23.167.194:37445)
0928 08:53:38.904427 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:38.904437 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:38.906257 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:38.906599 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:38.906604 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:38.907267 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:38.907511 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.907517 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.908497 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.908498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.911386 (+ 2888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.911389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.911823 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.911824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.911880 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.912229 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:38.912234 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:38.913297 (+ 1063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:38.914936 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.914937 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.914937 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.915110 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.915337 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.915337 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.915337 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.915385 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.915639 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:38.915639 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:38.915672 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 08:53:38.916134 (+ 462us) client_negotiation.cc:240] Negotiation successful
0928 08:53:38.916327 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":114,"thread_start_us":78,"threads_started":1}
I20260928 08:53:38.916577 31412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.904175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:40727 (local address 127.23.167.194:47423)
0928 08:53:38.904338 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:38.904379 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:38.906244 (+ 1865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:38.906599 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:38.906604 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:38.907266 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:38.907511 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.907517 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.908497 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.908498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.909875 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.909878 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.910376 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.910377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.910437 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.910935 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:38.910940 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:38.912933 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:38.914620 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.914627 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.914630 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.914894 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.915089 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.915089 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.915090 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.915208 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.915508 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:38.915515 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:38.915672 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 08:53:38.916134 (+ 462us) client_negotiation.cc:240] Negotiation successful
0928 08:53:38.916384 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":183,"thread_start_us":79,"threads_started":1}
I20260928 08:53:38.917016 31410 heartbeater.cc:347] Connected to a master server at 127.23.167.253:44651
I20260928 08:53:38.917125 31410 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:38.917227 31408 heartbeater.cc:347] Connected to a master server at 127.23.167.252:40727
I20260928 08:53:38.917266 31408 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:38.917309 31410 heartbeater.cc:511] Master 127.23.167.253:44651 requested a full tablet report, sending...
I20260928 08:53:38.917318 31408 heartbeater.cc:511] Master 127.23.167.252:40727 requested a full tablet report, sending...
I20260928 08:53:38.917712 31092 ts_manager.cc:194] Registered new tserver with Master: e75b3311a1d5482ea942df5c55632674 (127.23.167.194:38385)
I20260928 08:53:38.917712 30906 ts_manager.cc:194] Registered new tserver with Master: e75b3311a1d5482ea942df5c55632674 (127.23.167.194:38385)
I20260928 08:53:38.917833 30499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.904618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:59411 (local address 127.23.167.254:34791)
0928 08:53:38.904659 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:38.904661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:38.905422 (+ 761us) server_negotiation.cc:407] Connection header received
0928 08:53:38.906367 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:38.906370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:38.906383 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:38.906411 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:38.907613 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.908357 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.909885 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.910121 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.910182 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.914276 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:38.914278 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:38.914278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:38.914286 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:38.915817 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.917095 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.917097 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.917097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.917161 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:38.917347 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:38.917348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:38.917348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:38.917465 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:38.917508 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:38.917672 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 08:53:38.917745 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 08:53:38.918147 31414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:38.904444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.194:59411)
0928 08:53:38.905364 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:38.905373 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:38.906251 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:38.906600 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:38.906604 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:38.907236 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:38.907511 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.907516 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.908485 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.908486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.909814 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:38.909816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:38.910246 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:38.910247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:38.910364 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:38.910822 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:38.910832 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:38.913594 (+ 2762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:38.916793 (+ 3199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.916794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.916794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.916993 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.917233 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:38.917233 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:38.917234 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:38.917287 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:38.917534 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:38.917534 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:38.917573 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 08:53:38.917950 (+ 377us) client_negotiation.cc:240] Negotiation successful
0928 08:53:38.918048 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"mutex_wait_us":74,"thread_start_us":53,"threads_started":1}
I20260928 08:53:38.918511 31409 heartbeater.cc:347] Connected to a master server at 127.23.167.254:34791
I20260928 08:53:38.918558 31409 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:38.918615 31409 heartbeater.cc:511] Master 127.23.167.254:34791 requested a full tablet report, sending...
I20260928 08:53:38.918836 30306 ts_manager.cc:194] Re-registered known tserver with Master: e75b3311a1d5482ea942df5c55632674 (127.23.167.194:38385)
I20260928 08:53:38.919183 30306 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.167.194@KRBTEST.COM'} at 127.23.167.194:59411
W20260928 08:53:38.995193 31415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:53:38.995398 31415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:53:38.995428 31415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:53:38.996542 31415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:53:38.996579 31415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:53:38.996613 31415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:53:38.998037 31415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:36571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.195.keytab
--principal=oryx/127.23.167.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:41903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=44005
--tserver_master_addrs=127.23.167.254:34791,127.23.167.253:44651,127.23.167.252:40727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:53:38.998262 31415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:53:38.998531 31415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:53:39.003252 31421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:39.003372 31415 server_base.cc:1034] running on GCE node
W20260928 08:53:39.003275 31420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:53:39.003443 31423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:53:39.003646 31415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:53:39.003895 31415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:53:39.005051 31415 hybrid_clock.cc:648] HybridClock initialized: now 1790585619005035 us; error 29 us; skew 500 ppm
Sep 28 08:53:39 dist-test-slave-pbjp krb5kdc[30261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585619, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:53:39.007637 31415 init.cc:377] Logged in from keytab as oryx/127.23.167.195@KRBTEST.COM (short username oryx)
I20260928 08:53:39.008085 31415 webserver.cc:460] Webserver started at http://127.23.167.195:44005/ using document root <none> and password file <none>
I20260928 08:53:39.008298 31415 fs_manager.cc:359] Metadata directory not provided
I20260928 08:53:39.008358 31415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:53:39.009697 31415 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:53:39.010231 31430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:39.010363 31415 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:53:39.010442 31415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "f5265171404243129b6726d8707066be"
format_stamp: "Formatted at 2026-09-28 08:53:38 on dist-test-slave-pbjp"
server_key: "7c23fcd00ab2ec86ff681c7cb281ed0d"
server_key_iv: "db7dab64a08f601237317d763b80ba99"
server_key_version: "encryptionkey@0"
I20260928 08:53:39.010758 31415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:53:39.026084 31415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:53:39.027887 31415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:53:39.028017 31415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:53:39.028223 31415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:53:39.028515 31415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:53:39.028553 31415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:39.028581 31415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:53:39.028605 31415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:53:39.034077 31415 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:41903
I20260928 08:53:39.034121 31543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:41903 every 8 connection(s)
I20260928 08:53:39.034453 31415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
I20260928 08:53:39.036942 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 31415
Sep 28 08:53:39 dist-test-slave-pbjp krb5kdc[30261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585619, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.252@KRBTEST.COM
Sep 28 08:53:39 dist-test-slave-pbjp krb5kdc[30261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585619, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM
Sep 28 08:53:39 dist-test-slave-pbjp krb5kdc[30261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585619, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.253@KRBTEST.COM
I20260928 08:53:39.046768 31548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:39.035162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.195:38761)
0928 08:53:39.035348 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:39.035397 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:39.036419 (+ 1022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:39.036683 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:39.036684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:39.037146 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:39.037443 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:39.037449 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.038417 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.038418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:39.039736 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:39.039739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.040266 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.040267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:39.040409 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:39.040953 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:39.040962 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:39.043396 (+ 2434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:39.045202 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:39.045203 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:39.045203 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:39.045421 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:39.045627 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:39.045628 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:39.045628 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:39.045752 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:39.046057 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:39.046063 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:39.046230 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 08:53:39.046345 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 08:53:39.046523 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":7,"thread_start_us":85,"threads_started":1}
I20260928 08:53:39.047341 30499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:39.035148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:38761 (local address 127.23.167.254:34791)
0928 08:53:39.035198 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:39.035200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:39.035477 (+ 277us) server_negotiation.cc:407] Connection header received
0928 08:53:39.036598 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:39.036600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:39.036612 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:39.036662 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:39.037541 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.038281 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.039858 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.040141 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.040180 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:39.043486 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:39.043488 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:39.043488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:39.043495 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:39.045116 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:39.045500 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:39.045501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:39.045502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:39.045561 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:39.045818 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:39.045819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:39.045819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:39.045951 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:39.046010 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:39.047169 (+ 1159us) server_negotiation.cc:299] Negotiation successful
0928 08:53:39.047251 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 08:53:39.047473 31550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:39.035449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:40727 (local address 127.23.167.195:49653)
0928 08:53:39.035574 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:39.035583 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:39.036430 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:39.036810 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:39.036811 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:39.037147 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:39.037443 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:39.037449 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.039210 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.039211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:39.040032 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:39.040035 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.040541 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.040542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:39.040606 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:39.041100 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:39.041105 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:39.042783 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:39.044956 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:39.044963 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:39.044966 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:39.046561 (+ 1595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:39.046749 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:39.046750 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:39.046750 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:39.046806 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:39.047193 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:39.047194 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:39.047233 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 08:53:39.047285 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 08:53:39.047372 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":46,"thread_start_us":66,"threads_started":1}
I20260928 08:53:39.047741 31545 heartbeater.cc:347] Connected to a master server at 127.23.167.254:34791
I20260928 08:53:39.047837 31545 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:39.047927 31275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:39.035612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:49653 (local address 127.23.167.252:40727)
0928 08:53:39.035677 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:39.035679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:39.035688 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:53:39.036717 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:39.036718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:39.036730 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:39.036780 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:39.038386 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.039128 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.040119 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.040433 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.040487 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:39.042878 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:39.042879 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:39.042880 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:39.042887 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:39.044868 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:39.046632 (+ 1764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:39.046633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:39.046633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:39.046691 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:39.046981 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:39.046982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:39.046983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:39.047111 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:39.047165 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:39.047743 (+ 578us) server_negotiation.cc:299] Negotiation successful
0928 08:53:39.047833 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 08:53:39.048025 31545 heartbeater.cc:511] Master 127.23.167.254:34791 requested a full tablet report, sending...
I20260928 08:53:39.048432 30306 ts_manager.cc:194] Re-registered known tserver with Master: f5265171404243129b6726d8707066be (127.23.167.195:41903)
I20260928 08:53:39.048481 31544 heartbeater.cc:347] Connected to a master server at 127.23.167.252:40727
I20260928 08:53:39.048527 31544 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:39.048612 31544 heartbeater.cc:511] Master 127.23.167.252:40727 requested a full tablet report, sending...
I20260928 08:53:39.048888 30306 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.167.195@KRBTEST.COM'} at 127.23.167.195:38761
I20260928 08:53:39.048906 31092 ts_manager.cc:194] Registered new tserver with Master: f5265171404243129b6726d8707066be (127.23.167.195:41903)
I20260928 08:53:39.049774 30957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:39.035169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:41471 (local address 127.23.167.253:44651)
0928 08:53:39.035412 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:39.035414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:39.035478 (+ 64us) server_negotiation.cc:407] Connection header received
0928 08:53:39.036529 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:39.036531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:39.036543 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:39.036589 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:39.038370 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.039124 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.043182 (+ 4058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.043467 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.043519 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:39.046223 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:39.046225 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:39.046225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:39.046231 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:39.047771 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:39.048784 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:39.048785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:39.048786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:39.048847 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:39.049039 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:39.049040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:39.049040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:39.049188 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:39.049243 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:39.049588 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 08:53:39.049664 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212}
I20260928 08:53:39.050127 31549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:39.035162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:44651 (local address 127.23.167.195:41471)
0928 08:53:39.035348 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:39.035419 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:39.036424 (+ 1005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:39.036672 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:39.036676 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:39.038201 (+ 1525us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:53:39.038267 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:39.038269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.039210 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.039211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:39.043044 (+ 3833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:39.043048 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.043569 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.043570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:39.043632 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:39.044742 (+ 1110us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:53:39.044747 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:53:39.046138 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:53:39.048083 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:39.048085 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:39.048085 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:39.048303 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:39.048910 (+ 607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:53:39.048910 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:53:39.048911 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:53:39.048968 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:53:39.049273 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:53:39.049274 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:53:39.049328 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 08:53:39.049909 (+ 581us) client_negotiation.cc:240] Negotiation successful
0928 08:53:39.050022 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":4,"thread_start_us":90,"threads_started":1}
I20260928 08:53:39.050462 31546 heartbeater.cc:347] Connected to a master server at 127.23.167.253:44651
I20260928 08:53:39.050509 31546 heartbeater.cc:464] Registering TS with master...
I20260928 08:53:39.050572 31546 heartbeater.cc:511] Master 127.23.167.253:44651 requested a full tablet report, sending...
I20260928 08:53:39.050909 30906 ts_manager.cc:194] Registered new tserver with Master: f5265171404243129b6726d8707066be (127.23.167.195:41903)
I20260928 08:53:39.051196 31555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:39.038051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47742 (local address 127.23.167.254:34791)
0928 08:53:39.038943 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:39.038944 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:39.038951 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:53:39.038970 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:39.038971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:39.038982 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:39.039039 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:39.039826 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.040540 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.041532 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.041845 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.041903 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:39.045279 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:39.045281 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:39.045281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:39.045288 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:39.048685 (+ 3397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:39.049410 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:39.049411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:39.049411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:39.049471 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:39.050434 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:39.050435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:39.050436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:39.050569 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:39.050660 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:39.050941 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 08:53:39.051108 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":864,"thread_start_us":80,"threads_started":1}
I20260928 08:53:39.051748 31559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:39.038638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50726 (local address 127.23.167.252:40727)
0928 08:53:39.038881 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:39.038883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:39.039730 (+ 847us) server_negotiation.cc:407] Connection header received
0928 08:53:39.039753 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:39.039754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:39.039767 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:39.039814 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:39.040482 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.041656 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.045150 (+ 3494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.045433 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.045476 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:39.047645 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:39.047647 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:39.047647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:39.047654 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:39.050529 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:39.050914 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:39.050915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:39.050915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:39.050974 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:39.051184 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:39.051185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:39.051185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:39.051314 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:39.051390 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:39.051569 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 08:53:39.051652 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":42,"threads_started":1}
I20260928 08:53:39.053212 31557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:39.038288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49362 (local address 127.23.167.253:44651)
0928 08:53:39.039553 (+ 1265us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:39.039555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:39.040759 (+ 1204us) server_negotiation.cc:407] Connection header received
0928 08:53:39.040823 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:39.040824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:39.040837 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:39.041272 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:39.042100 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.042779 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.046866 (+ 4087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.047170 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.047229 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:39.050263 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:39.050264 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:39.050265 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:39.050272 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:39.052175 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:39.052507 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:39.052508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:39.052508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:39.052558 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:39.052734 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:39.052735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:39.052736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:39.052840 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:39.052921 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:39.053026 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 08:53:39.053124 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1246,"thread_start_us":60,"threads_started":1}
I20260928 08:53:39.052718 30306 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:53:39.054565 30306 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:53:39.061863 31569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:39.058288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57774 (local address 127.23.167.195:41903)
0928 08:53:39.058498 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:39.058501 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:39.058512 (+ 11us) server_negotiation.cc:407] Connection header received
0928 08:53:39.058591 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:39.058593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:39.058606 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:39.058668 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:53:39.058909 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.059581 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.061004 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.061539 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.061625 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:39.061687 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:39.061752 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 08:53:39.061789 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":73,"threads_started":1}
I20260928 08:53:39.061923 30865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:39.057920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:38385 (local address 127.0.0.1:57756)
0928 08:53:39.058051 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:39.058063 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:39.058107 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:39.058387 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:39.058388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:39.058390 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:53:39.058432 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:39.058434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.059799 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.059799 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:39.060800 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:39.060802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.061670 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.061671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:39.061751 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:39.061756 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 08:53:39.061795 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 08:53:39.061828 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":15}
I20260928 08:53:39.061931 31568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:39.057948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:38813 (local address 127.0.0.1:44724)
0928 08:53:39.058271 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:39.058279 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:39.058358 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:39.058575 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:39.058576 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:39.058577 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:53:39.058614 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:39.058615 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.059962 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.059963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:39.060735 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:39.060738 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.061670 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.061671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:39.061751 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:39.061757 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 08:53:39.061786 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 08:53:39.061825 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":8,"thread_start_us":107,"threads_started":1}
I20260928 08:53:39.062052 31565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:39.058135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:41903 (local address 127.0.0.1:57774)
0928 08:53:39.058474 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:39.058483 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:39.058522 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:39.058679 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:39.058680 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:39.058681 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:53:39.058850 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:39.058852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.059657 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.059658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:39.060940 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:39.060942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.061615 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.061616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:39.061680 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:39.061685 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 08:53:39.061737 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 08:53:39.061781 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":1,"thread_start_us":57,"threads_started":1}
I20260928 08:53:39.062119 31567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:39.058064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44724 (local address 127.23.167.193:38813)
0928 08:53:39.058220 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:39.058226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:39.058335 (+ 109us) server_negotiation.cc:407] Connection header received
0928 08:53:39.058457 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:39.058460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:39.058477 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:39.058735 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:53:39.058795 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.059896 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.060819 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.061570 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.061903 (+ 333us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:39.061966 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:39.061993 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 08:53:39.062032 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20260928 08:53:39.062723 31478 tablet_service.cc:1467] Processing CreateTablet for tablet dd5a53b09e654d94b80dd3aba0a535d0 (DEFAULT_TABLE table=test-table [id=06c82292a1ae4713bd1719ce9932c218]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:53:39.062795 31205 tablet_service.cc:1467] Processing CreateTablet for tablet dd5a53b09e654d94b80dd3aba0a535d0 (DEFAULT_TABLE table=test-table [id=06c82292a1ae4713bd1719ce9932c218]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:53:39.063098 31478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd5a53b09e654d94b80dd3aba0a535d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:39.063136 31205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd5a53b09e654d94b80dd3aba0a535d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:39.063333 31566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:39.058012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57756 (local address 127.23.167.194:38385)
0928 08:53:39.058225 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:39.058228 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:39.058237 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:53:39.058276 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:39.058279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:39.058296 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:39.058376 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:53:39.058584 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.059721 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.060869 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.061570 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.062773 (+ 1203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:39.063049 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:39.063139 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 08:53:39.063256 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":92,"threads_started":1}
I20260928 08:53:39.064110 31342 tablet_service.cc:1467] Processing CreateTablet for tablet dd5a53b09e654d94b80dd3aba0a535d0 (DEFAULT_TABLE table=test-table [id=06c82292a1ae4713bd1719ce9932c218]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:53:39.064435 31342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd5a53b09e654d94b80dd3aba0a535d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:53:39.065881 31570 tablet_bootstrap.cc:492] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418: Bootstrap starting.
I20260928 08:53:39.066632 31570 tablet_bootstrap.cc:654] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:39.066695 31571 tablet_bootstrap.cc:492] T dd5a53b09e654d94b80dd3aba0a535d0 P f5265171404243129b6726d8707066be: Bootstrap starting.
I20260928 08:53:39.067005 31570 log.cc:824] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:39.067268 31571 tablet_bootstrap.cc:654] T dd5a53b09e654d94b80dd3aba0a535d0 P f5265171404243129b6726d8707066be: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:39.067490 31571 log.cc:824] T dd5a53b09e654d94b80dd3aba0a535d0 P f5265171404243129b6726d8707066be: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:39.067657 31573 tablet_bootstrap.cc:492] T dd5a53b09e654d94b80dd3aba0a535d0 P e75b3311a1d5482ea942df5c55632674: Bootstrap starting.
I20260928 08:53:39.067981 31570 tablet_bootstrap.cc:492] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418: No bootstrap required, opened a new log
I20260928 08:53:39.068051 31570 ts_tablet_manager.cc:1397] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:53:39.068220 31573 tablet_bootstrap.cc:654] T dd5a53b09e654d94b80dd3aba0a535d0 P e75b3311a1d5482ea942df5c55632674: Neither blocks nor log segments found. Creating new log.
I20260928 08:53:39.068466 31573 log.cc:824] T dd5a53b09e654d94b80dd3aba0a535d0 P e75b3311a1d5482ea942df5c55632674: Log is configured to *not* fsync() on all Append() calls
I20260928 08:53:39.069231 31573 tablet_bootstrap.cc:492] T dd5a53b09e654d94b80dd3aba0a535d0 P e75b3311a1d5482ea942df5c55632674: No bootstrap required, opened a new log
I20260928 08:53:39.069288 31573 ts_tablet_manager.cc:1397] T dd5a53b09e654d94b80dd3aba0a535d0 P e75b3311a1d5482ea942df5c55632674: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:53:39.069620 31571 tablet_bootstrap.cc:492] T dd5a53b09e654d94b80dd3aba0a535d0 P f5265171404243129b6726d8707066be: No bootstrap required, opened a new log
I20260928 08:53:39.069694 31571 ts_tablet_manager.cc:1397] T dd5a53b09e654d94b80dd3aba0a535d0 P f5265171404243129b6726d8707066be: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 08:53:39.070616 31570 raft_consensus.cc:348] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75b3311a1d5482ea942df5c55632674" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 38385 } } peers { permanent_uuid: "513c83afb10d4d09b2f13357edabd418" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 38813 } } peers { permanent_uuid: "f5265171404243129b6726d8707066be" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 41903 } }
I20260928 08:53:39.070788 31570 raft_consensus.cc:374] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:39.070823 31570 raft_consensus.cc:729] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 513c83afb10d4d09b2f13357edabd418, State: Initialized, Role: FOLLOWER
I20260928 08:53:39.070996 31570 consensus_queue.cc:260] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75b3311a1d5482ea942df5c55632674" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 38385 } } peers { permanent_uuid: "513c83afb10d4d09b2f13357edabd418" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 38813 } } peers { permanent_uuid: "f5265171404243129b6726d8707066be" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 41903 } }
I20260928 08:53:39.071269 31570 ts_tablet_manager.cc:1428] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:53:39.071276 31573 raft_consensus.cc:348] T dd5a53b09e654d94b80dd3aba0a535d0 P e75b3311a1d5482ea942df5c55632674 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75b3311a1d5482ea942df5c55632674" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 38385 } } peers { permanent_uuid: "513c83afb10d4d09b2f13357edabd418" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 38813 } } peers { permanent_uuid: "f5265171404243129b6726d8707066be" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 41903 } }
I20260928 08:53:39.071426 31272 heartbeater.cc:503] Master 127.23.167.254:34791 was elected leader, sending a full tablet report...
I20260928 08:53:39.071465 31573 raft_consensus.cc:374] T dd5a53b09e654d94b80dd3aba0a535d0 P e75b3311a1d5482ea942df5c55632674 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:39.071503 31573 raft_consensus.cc:729] T dd5a53b09e654d94b80dd3aba0a535d0 P e75b3311a1d5482ea942df5c55632674 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e75b3311a1d5482ea942df5c55632674, State: Initialized, Role: FOLLOWER
I20260928 08:53:39.071601 31573 consensus_queue.cc:260] T dd5a53b09e654d94b80dd3aba0a535d0 P e75b3311a1d5482ea942df5c55632674 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75b3311a1d5482ea942df5c55632674" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 38385 } } peers { permanent_uuid: "513c83afb10d4d09b2f13357edabd418" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 38813 } } peers { permanent_uuid: "f5265171404243129b6726d8707066be" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 41903 } }
I20260928 08:53:39.071835 31573 ts_tablet_manager.cc:1428] T dd5a53b09e654d94b80dd3aba0a535d0 P e75b3311a1d5482ea942df5c55632674: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:53:39.071950 31409 heartbeater.cc:503] Master 127.23.167.254:34791 was elected leader, sending a full tablet report...
I20260928 08:53:39.072160 31571 raft_consensus.cc:348] T dd5a53b09e654d94b80dd3aba0a535d0 P f5265171404243129b6726d8707066be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75b3311a1d5482ea942df5c55632674" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 38385 } } peers { permanent_uuid: "513c83afb10d4d09b2f13357edabd418" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 38813 } } peers { permanent_uuid: "f5265171404243129b6726d8707066be" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 41903 } }
I20260928 08:53:39.072309 31571 raft_consensus.cc:374] T dd5a53b09e654d94b80dd3aba0a535d0 P f5265171404243129b6726d8707066be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:53:39.072345 31571 raft_consensus.cc:729] T dd5a53b09e654d94b80dd3aba0a535d0 P f5265171404243129b6726d8707066be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5265171404243129b6726d8707066be, State: Initialized, Role: FOLLOWER
I20260928 08:53:39.073155 31571 consensus_queue.cc:260] T dd5a53b09e654d94b80dd3aba0a535d0 P f5265171404243129b6726d8707066be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75b3311a1d5482ea942df5c55632674" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 38385 } } peers { permanent_uuid: "513c83afb10d4d09b2f13357edabd418" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 38813 } } peers { permanent_uuid: "f5265171404243129b6726d8707066be" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 41903 } }
I20260928 08:53:39.073414 31571 ts_tablet_manager.cc:1428] T dd5a53b09e654d94b80dd3aba0a535d0 P f5265171404243129b6726d8707066be: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 08:53:39.073446 31545 heartbeater.cc:503] Master 127.23.167.254:34791 was elected leader, sending a full tablet report...
I20260928 08:53:39.118050 31576 raft_consensus.cc:482] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:53:39.118184 31576 raft_consensus.cc:504] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75b3311a1d5482ea942df5c55632674" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 38385 } } peers { permanent_uuid: "513c83afb10d4d09b2f13357edabd418" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 38813 } } peers { permanent_uuid: "f5265171404243129b6726d8707066be" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 41903 } }
I20260928 08:53:39.118556 31576 leader_election.cc:288] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e75b3311a1d5482ea942df5c55632674 (127.23.167.194:38385), f5265171404243129b6726d8707066be (127.23.167.195:41903)
I20260928 08:53:39.121589 31566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:39.118816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:42655 (local address 127.23.167.194:38385)
0928 08:53:39.118884 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:39.118886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:39.119030 (+ 144us) server_negotiation.cc:407] Connection header received
0928 08:53:39.119084 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:39.119085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:39.119096 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:39.119149 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:53:39.119307 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.119910 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.120855 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.121280 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.121333 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:39.121343 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:39.121473 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 08:53:39.121516 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 08:53:39.121587 31276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:39.118742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:38385 (local address 127.23.167.193:42655)
0928 08:53:39.118974 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:39.118984 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:39.119031 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:39.119150 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:39.119151 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:39.119152 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:53:39.119215 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:39.119216 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.119992 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.119993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:39.120768 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:39.120770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.121344 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.121345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:39.121418 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:39.121422 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:53:39.121467 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 08:53:39.121499 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":181}
I20260928 08:53:39.121637 31569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:39.118831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:36303 (local address 127.23.167.195:41903)
0928 08:53:39.118874 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:39.118876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:39.118886 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:53:39.118921 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:39.118922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:39.118929 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:39.119008 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:53:39.119252 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.119882 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.120897 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.121294 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.121361 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:39.121395 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:39.121542 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 08:53:39.121580 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 08:53:39.121815 31277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:39.118742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:41903 (local address 127.23.167.193:36303)
0928 08:53:39.118796 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:39.118808 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:53:39.118853 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:53:39.119013 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:53:39.119013 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:53:39.119015 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:53:39.119125 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:39.119127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.119958 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.119960 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:39.120767 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:53:39.120770 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.121368 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.121369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:53:39.121475 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:39.121479 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:53:39.121530 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 08:53:39.121569 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":24}
I20260928 08:53:39.121985 31362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5a53b09e654d94b80dd3aba0a535d0" candidate_uuid: "513c83afb10d4d09b2f13357edabd418" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e75b3311a1d5482ea942df5c55632674" is_pre_election: true
I20260928 08:53:39.122139 31362 raft_consensus.cc:2454] T dd5a53b09e654d94b80dd3aba0a535d0 P e75b3311a1d5482ea942df5c55632674 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 513c83afb10d4d09b2f13357edabd418 in term 0.
I20260928 08:53:39.122113 31498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5a53b09e654d94b80dd3aba0a535d0" candidate_uuid: "513c83afb10d4d09b2f13357edabd418" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5265171404243129b6726d8707066be" is_pre_election: true
I20260928 08:53:39.122255 31498 raft_consensus.cc:2454] T dd5a53b09e654d94b80dd3aba0a535d0 P f5265171404243129b6726d8707066be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 513c83afb10d4d09b2f13357edabd418 in term 0.
I20260928 08:53:39.122345 31159 leader_election.cc:302] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 513c83afb10d4d09b2f13357edabd418, e75b3311a1d5482ea942df5c55632674; no voters:
I20260928 08:53:39.122484 31576 raft_consensus.cc:2781] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:53:39.122553 31576 raft_consensus.cc:482] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:53:39.122582 31576 raft_consensus.cc:3037] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:39.123370 31576 raft_consensus.cc:504] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75b3311a1d5482ea942df5c55632674" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 38385 } } peers { permanent_uuid: "513c83afb10d4d09b2f13357edabd418" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 38813 } } peers { permanent_uuid: "f5265171404243129b6726d8707066be" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 41903 } }
I20260928 08:53:39.123503 31576 leader_election.cc:288] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418 [CANDIDATE]: Term 1 election: Requested vote from peers e75b3311a1d5482ea942df5c55632674 (127.23.167.194:38385), f5265171404243129b6726d8707066be (127.23.167.195:41903)
I20260928 08:53:39.123646 31362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5a53b09e654d94b80dd3aba0a535d0" candidate_uuid: "513c83afb10d4d09b2f13357edabd418" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e75b3311a1d5482ea942df5c55632674"
I20260928 08:53:39.123654 31498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5a53b09e654d94b80dd3aba0a535d0" candidate_uuid: "513c83afb10d4d09b2f13357edabd418" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5265171404243129b6726d8707066be"
I20260928 08:53:39.123728 31498 raft_consensus.cc:3037] T dd5a53b09e654d94b80dd3aba0a535d0 P f5265171404243129b6726d8707066be [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:39.123728 31362 raft_consensus.cc:3037] T dd5a53b09e654d94b80dd3aba0a535d0 P e75b3311a1d5482ea942df5c55632674 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:53:39.124586 31498 raft_consensus.cc:2454] T dd5a53b09e654d94b80dd3aba0a535d0 P f5265171404243129b6726d8707066be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 513c83afb10d4d09b2f13357edabd418 in term 1.
I20260928 08:53:39.124586 31362 raft_consensus.cc:2454] T dd5a53b09e654d94b80dd3aba0a535d0 P e75b3311a1d5482ea942df5c55632674 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 513c83afb10d4d09b2f13357edabd418 in term 1.
I20260928 08:53:39.124759 31158 leader_election.cc:302] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 513c83afb10d4d09b2f13357edabd418, f5265171404243129b6726d8707066be; no voters:
I20260928 08:53:39.124848 31576 raft_consensus.cc:2781] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:53:39.124989 31576 raft_consensus.cc:686] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418 [term 1 LEADER]: Becoming Leader. State: Replica: 513c83afb10d4d09b2f13357edabd418, State: Running, Role: LEADER
I20260928 08:53:39.125154 31576 consensus_queue.cc:237] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75b3311a1d5482ea942df5c55632674" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 38385 } } peers { permanent_uuid: "513c83afb10d4d09b2f13357edabd418" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 38813 } } peers { permanent_uuid: "f5265171404243129b6726d8707066be" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 41903 } }
I20260928 08:53:39.126015 30305 catalog_manager.cc:5495] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418 reported cstate change: term changed from 0 to 1, leader changed from <none> to 513c83afb10d4d09b2f13357edabd418 (127.23.167.193). New cstate: current_term: 1 leader_uuid: "513c83afb10d4d09b2f13357edabd418" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75b3311a1d5482ea942df5c55632674" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 38385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "513c83afb10d4d09b2f13357edabd418" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 38813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5265171404243129b6726d8707066be" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 41903 } health_report { overall_health: UNKNOWN } } }
I20260928 08:53:39.138914 31567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:39.136426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44740 (local address 127.23.167.193:38813)
0928 08:53:39.136468 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:39.136470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:39.136509 (+ 39us) server_negotiation.cc:407] Connection header received
0928 08:53:39.136571 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:39.136572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:39.136590 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:39.136653 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:53:39.136772 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.137349 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.138185 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.138442 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.138506 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:39.138626 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:53:39.138727 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:53:39.138783 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:39.138796 (+ 13us) server_negotiation.cc:299] Negotiation successful
0928 08:53:39.138813 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 08:53:39.140815 31362 raft_consensus.cc:1261] T dd5a53b09e654d94b80dd3aba0a535d0 P e75b3311a1d5482ea942df5c55632674 [term 1 FOLLOWER]: Refusing update from remote peer 513c83afb10d4d09b2f13357edabd418: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:53:39.140810 31498 raft_consensus.cc:1261] T dd5a53b09e654d94b80dd3aba0a535d0 P f5265171404243129b6726d8707066be [term 1 FOLLOWER]: Refusing update from remote peer 513c83afb10d4d09b2f13357edabd418: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:53:39.141158 31576 consensus_queue.cc:1035] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e75b3311a1d5482ea942df5c55632674" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 38385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:53:39.141340 31579 consensus_queue.cc:1035] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5265171404243129b6726d8707066be" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 41903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:53:39.143848 31582 mvcc.cc:205] Tried to move back new op lower bound from 7334238695997816832 to 7334238695937265664. Current Snapshot: MvccSnapshot[applied={T|T < 7334238695997816832}]
I20260928 08:53:39.145931 31583 mvcc.cc:205] Tried to move back new op lower bound from 7334238695997816832 to 7334238695937265664. Current Snapshot: MvccSnapshot[applied={T|T < 7334238695997816832}]
I20260928 08:53:39.146785 30305 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:53:39.146869 30305 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:53:39.148396 30305 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b528077fc484f4fae48815cbca7e762: Sending DeleteTablet for 3 replicas of tablet dd5a53b09e654d94b80dd3aba0a535d0
I20260928 08:53:39.148802 31342 tablet_service.cc:1514] Processing DeleteTablet for tablet dd5a53b09e654d94b80dd3aba0a535d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:53:39 UTC) from {username='oryx', principal='oryx/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:57756
I20260928 08:53:39.148818 31478 tablet_service.cc:1514] Processing DeleteTablet for tablet dd5a53b09e654d94b80dd3aba0a535d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:53:39 UTC) from {username='oryx', principal='oryx/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:57774
I20260928 08:53:39.148900 31205 tablet_service.cc:1514] Processing DeleteTablet for tablet dd5a53b09e654d94b80dd3aba0a535d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:53:39 UTC) from {username='oryx', principal='oryx/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:44724
I20260928 08:53:39.149171 31593 tablet_replica.cc:330] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418: stopping tablet replica
I20260928 08:53:39.149281 31593 raft_consensus.cc:2229] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:53:39.149443 31593 raft_consensus.cc:2258] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:53:39.150044 31593 ts_tablet_manager.cc:1905] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:53:39.150321 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 30274
I20260928 08:53:39.152107 31593 ts_tablet_manager.cc:1918] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:53:39.152184 31593 log.cc:1196] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-0/wal/wals/dd5a53b09e654d94b80dd3aba0a535d0
I20260928 08:53:39.152436 31593 ts_tablet_manager.cc:1939] T dd5a53b09e654d94b80dd3aba0a535d0 P 513c83afb10d4d09b2f13357edabd418: Deleting consensus metadata
I20260928 08:53:39.152844 31591 tablet_replica.cc:330] T dd5a53b09e654d94b80dd3aba0a535d0 P e75b3311a1d5482ea942df5c55632674: stopping tablet replica
I20260928 08:53:39.152967 31591 raft_consensus.cc:2229] T dd5a53b09e654d94b80dd3aba0a535d0 P e75b3311a1d5482ea942df5c55632674 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:53:39.153108 31591 raft_consensus.cc:2258] T dd5a53b09e654d94b80dd3aba0a535d0 P e75b3311a1d5482ea942df5c55632674 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:53:39.153297 31592 tablet_replica.cc:330] T dd5a53b09e654d94b80dd3aba0a535d0 P f5265171404243129b6726d8707066be: stopping tablet replica
I20260928 08:53:39.153410 31592 raft_consensus.cc:2229] T dd5a53b09e654d94b80dd3aba0a535d0 P f5265171404243129b6726d8707066be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:53:39.153512 31592 raft_consensus.cc:2258] T dd5a53b09e654d94b80dd3aba0a535d0 P f5265171404243129b6726d8707066be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:53:39.153641 31591 ts_tablet_manager.cc:1905] T dd5a53b09e654d94b80dd3aba0a535d0 P e75b3311a1d5482ea942df5c55632674: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:53:39.154008 31592 ts_tablet_manager.cc:1905] T dd5a53b09e654d94b80dd3aba0a535d0 P f5265171404243129b6726d8707066be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:53:39.155656 31592 ts_tablet_manager.cc:1918] T dd5a53b09e654d94b80dd3aba0a535d0 P f5265171404243129b6726d8707066be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:53:39.155682 31591 ts_tablet_manager.cc:1918] T dd5a53b09e654d94b80dd3aba0a535d0 P e75b3311a1d5482ea942df5c55632674: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:53:39.155723 31592 log.cc:1196] T dd5a53b09e654d94b80dd3aba0a535d0 P f5265171404243129b6726d8707066be: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-2/wal/wals/dd5a53b09e654d94b80dd3aba0a535d0
I20260928 08:53:39.155750 31591 log.cc:1196] T dd5a53b09e654d94b80dd3aba0a535d0 P e75b3311a1d5482ea942df5c55632674: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0/minicluster-data/ts-1/wal/wals/dd5a53b09e654d94b80dd3aba0a535d0
I20260928 08:53:39.155989 31592 ts_tablet_manager.cc:1939] T dd5a53b09e654d94b80dd3aba0a535d0 P f5265171404243129b6726d8707066be: Deleting consensus metadata
I20260928 08:53:39.155993 31591 ts_tablet_manager.cc:1939] T dd5a53b09e654d94b80dd3aba0a535d0 P e75b3311a1d5482ea942df5c55632674: Deleting consensus metadata
W20260928 08:53:39.157226 31431 connection.cc:535] server connection from 127.0.0.1:57774 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 08:53:39.157267 31160 connection.cc:535] server connection from 127.0.0.1:44724 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 08:53:39.157591 31297 connection.cc:535] server connection from 127.0.0.1:57756 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 08:53:39.161626 31559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:39.149751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50736 (local address 127.23.167.252:40727)
0928 08:53:39.149794 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:39.149796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:39.149934 (+ 138us) server_negotiation.cc:407] Connection header received
0928 08:53:39.150052 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:39.150053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:39.150068 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:39.150096 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:39.151711 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.152457 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.156434 (+ 3977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.156713 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.156757 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:39.158514 (+ 1757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:39.158516 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:39.158516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:39.158525 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:39.160150 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:39.160531 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:39.160532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:39.160533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:39.160592 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:39.160770 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:39.160771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:39.160771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:39.160905 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:39.160992 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:39.161435 (+ 443us) server_negotiation.cc:299] Negotiation successful
0928 08:53:39.161536 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 08:53:39.166591 31557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:39.159333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49376 (local address 127.23.167.253:44651)
0928 08:53:39.159380 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:39.159381 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:39.159744 (+ 363us) server_negotiation.cc:407] Connection header received
0928 08:53:39.159842 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:39.159843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:39.159857 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:39.159886 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:39.160574 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.161349 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.162329 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.162564 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.162596 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:39.163840 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:39.163841 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:39.163841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:39.163848 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:39.165504 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:39.165820 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:39.165822 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:39.165822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:39.165887 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:39.166046 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:39.166048 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:39.166048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:39.166175 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:39.166285 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:39.166400 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 08:53:39.166514 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 08:53:39.166785 31275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:39.159480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50740 (local address 127.23.167.252:40727)
0928 08:53:39.159521 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 08:53:39.159523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:53:39.159826 (+ 303us) server_negotiation.cc:407] Connection header received
0928 08:53:39.159923 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:53:39.159924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:53:39.159934 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:53:39.159962 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:53:39.160602 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.161416 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.162462 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:53:39.162752 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:53:39.162801 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:53:39.164262 (+ 1461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:53:39.164264 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:53:39.164264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:53:39.164271 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:53:39.165691 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:39.165986 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:39.165987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:39.165987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:39.166036 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:53:39.166229 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:53:39.166230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:53:39.166230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:53:39.166346 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:53:39.166421 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 08:53:39.166612 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 08:53:39.166685 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 08:53:40.145632 31602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:40.145190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.193:54037)
0928 08:53:40.145490 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:40.145578 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":87,"threads_started":1}
I20260928 08:53:40.145665 31603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:40.145380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.194:47627)
0928 08:53:40.145541 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:40.145622 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":96,"threads_started":1}
W20260928 08:53:40.145843 31272 heartbeater.cc:644] Failed to heartbeat to 127.23.167.254:34791 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
W20260928 08:53:40.145867 31409 heartbeater.cc:644] Failed to heartbeat to 127.23.167.254:34791 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
I20260928 08:53:40.146550 31604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:40.146259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.195:40597)
0928 08:53:40.146426 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:40.146505 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":90,"threads_started":1}
W20260928 08:53:40.146759 31545 heartbeater.cc:644] Failed to heartbeat to 127.23.167.254:34791 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
I20260928 08:53:40.649550 31605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b528077fc484f4fae48815cbca7e762)
I20260928 08:53:40.649693 31605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 44651 } attrs { promote: false } } peers { permanent_uuid: "ce49edb20df84405acc0d56445850999" member_type: NON_VOTER last_known_addr { host: "127.23.167.252" port: 40727 } attrs { promote: true } }
I20260928 08:53:40.649920 31605 leader_election.cc:288] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b528077fc484f4fae48815cbca7e762 (127.23.167.254:34791)
I20260928 08:53:40.650250 31606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:40.650062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.0.0.1:48182)
0928 08:53:40.650162 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:40.650211 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":47,"threads_started":1}
W20260928 08:53:40.650416 30891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
I20260928 08:53:40.650784 31606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:40.650687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.0.0.1:48196)
0928 08:53:40.650736 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:40.650763 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 08:53:40.650872 30891 leader_election.cc:334] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0b528077fc484f4fae48815cbca7e762 (127.23.167.254:34791): Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
I20260928 08:53:40.650928 30891 leader_election.cc:302] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 940646d9191e48d3903cb7037d8ed33c; no voters: 0b528077fc484f4fae48815cbca7e762
I20260928 08:53:40.651015 31605 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:53:41.146401 31608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:41.146158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.193:47815)
0928 08:53:41.146298 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:41.146342 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260928 08:53:41.146502 31609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:41.146265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.194:48679)
0928 08:53:41.146375 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:41.146412 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":68,"threads_started":1}
I20260928 08:53:41.147313 31610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:41.147117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.195:38357)
0928 08:53:41.147239 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:41.147269 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":51,"threads_started":1}
I20260928 08:53:42.147158 31611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:42.146891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.194:56529)
0928 08:53:42.147051 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:42.147100 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":66,"threads_started":1}
I20260928 08:53:42.147442 31612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:42.147097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.193:51151)
0928 08:53:42.147311 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:42.147354 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":62,"threads_started":1}
I20260928 08:53:42.147994 31613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:42.147763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.195:60679)
0928 08:53:42.147911 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:42.147947 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
I20260928 08:53:42.530210 31614 raft_consensus.cc:482] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b528077fc484f4fae48815cbca7e762)
I20260928 08:53:42.530316 31614 raft_consensus.cc:504] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 44651 } attrs { promote: false } } peers { permanent_uuid: "ce49edb20df84405acc0d56445850999" member_type: NON_VOTER last_known_addr { host: "127.23.167.252" port: 40727 } attrs { promote: true } }
I20260928 08:53:42.530452 31614 leader_election.cc:288] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b528077fc484f4fae48815cbca7e762 (127.23.167.254:34791)
I20260928 08:53:42.530827 31615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:42.530626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.0.0.1:48444)
0928 08:53:42.530746 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:42.530786 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":64,"threads_started":1}
I20260928 08:53:42.531250 31615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:42.531155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.0.0.1:48446)
0928 08:53:42.531199 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:42.531225 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 08:53:42.531320 30891 leader_election.cc:334] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0b528077fc484f4fae48815cbca7e762 (127.23.167.254:34791): Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
I20260928 08:53:42.531363 30891 leader_election.cc:302] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 940646d9191e48d3903cb7037d8ed33c; no voters: 0b528077fc484f4fae48815cbca7e762
I20260928 08:53:42.531454 31614 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:53:43.147957 31617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:43.147728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.194:35829)
0928 08:53:43.147859 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:43.147910 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260928 08:53:43.148217 31618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:43.147910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.193:58397)
0928 08:53:43.148138 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:43.148173 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":47,"threads_started":1}
I20260928 08:53:43.148614 31619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:43.148430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.195:36645)
0928 08:53:43.148532 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:43.148571 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":38,"threads_started":1}
I20260928 08:53:44.148660 31620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:44.148394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.194:37013)
0928 08:53:44.148528 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:44.148574 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":68,"threads_started":1}
I20260928 08:53:44.149245 31621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:44.149020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.195:48727)
0928 08:53:44.149152 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:44.149196 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
I20260928 08:53:44.149501 31622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:44.149304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.193:42657)
0928 08:53:44.149428 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:44.149459 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260928 08:53:45.149780 31623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:45.149371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.194:36321)
0928 08:53:45.149622 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:45.149720 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":75,"threads_started":1}
I20260928 08:53:45.150050 31625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:45.149848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.193:34963)
0928 08:53:45.149975 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:45.150008 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":51,"threads_started":1}
I20260928 08:53:45.150175 31624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:45.149683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.195:42933)
0928 08:53:45.150083 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:45.150124 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":92,"threads_started":1}
I20260928 08:53:45.174259 31626 raft_consensus.cc:482] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b528077fc484f4fae48815cbca7e762)
I20260928 08:53:45.174363 31626 raft_consensus.cc:504] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 44651 } attrs { promote: false } } peers { permanent_uuid: "ce49edb20df84405acc0d56445850999" member_type: NON_VOTER last_known_addr { host: "127.23.167.252" port: 40727 } attrs { promote: true } }
I20260928 08:53:45.174535 31626 leader_election.cc:288] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b528077fc484f4fae48815cbca7e762 (127.23.167.254:34791)
I20260928 08:53:45.174875 31627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:45.174675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.0.0.1:59524)
0928 08:53:45.174788 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:45.174835 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":48,"threads_started":1}
I20260928 08:53:45.175272 31627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:45.175190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.0.0.1:59534)
0928 08:53:45.175230 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:45.175255 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 08:53:45.175343 30891 leader_election.cc:334] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0b528077fc484f4fae48815cbca7e762 (127.23.167.254:34791): Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
I20260928 08:53:45.175452 30891 leader_election.cc:302] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 940646d9191e48d3903cb7037d8ed33c; no voters: 0b528077fc484f4fae48815cbca7e762
I20260928 08:53:45.175546 31626 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:53:46.150714 31631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:46.150533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.195:33145)
0928 08:53:46.150634 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:46.150663 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":42,"threads_started":1}
I20260928 08:53:46.150714 31630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:46.150444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.193:46721)
0928 08:53:46.150623 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:46.150663 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":84,"threads_started":1}
I20260928 08:53:46.150720 31629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:46.150443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.194:33999)
0928 08:53:46.150630 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:46.150671 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":63,"threads_started":1}
I20260928 08:53:47.151350 31632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:47.151066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.193:43595)
0928 08:53:47.151207 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:47.151289 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260928 08:53:47.151459 31633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:47.151225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.195:51403)
0928 08:53:47.151370 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:47.151411 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":65,"threads_started":1}
I20260928 08:53:47.151508 31634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:47.151279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.194:52981)
0928 08:53:47.151444 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:47.151471 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":54,"threads_started":1}
I20260928 08:53:47.386763 31635 raft_consensus.cc:482] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b528077fc484f4fae48815cbca7e762)
I20260928 08:53:47.386883 31635 raft_consensus.cc:504] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 44651 } attrs { promote: false } } peers { permanent_uuid: "ce49edb20df84405acc0d56445850999" member_type: NON_VOTER last_known_addr { host: "127.23.167.252" port: 40727 } attrs { promote: true } }
I20260928 08:53:47.387032 31635 leader_election.cc:288] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b528077fc484f4fae48815cbca7e762 (127.23.167.254:34791)
I20260928 08:53:47.387372 31636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:47.387184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.0.0.1:59680)
0928 08:53:47.387291 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:47.387332 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":42,"threads_started":1}
W20260928 08:53:47.387457 30891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20260928 08:53:47.387928 31636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:47.387809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.0.0.1:59684)
0928 08:53:47.387862 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:47.387890 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 08:53:47.388024 30891 leader_election.cc:334] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0b528077fc484f4fae48815cbca7e762 (127.23.167.254:34791): Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
I20260928 08:53:47.388072 30891 leader_election.cc:302] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 940646d9191e48d3903cb7037d8ed33c; no voters: 0b528077fc484f4fae48815cbca7e762
I20260928 08:53:47.388134 31635 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:53:48.152107 31638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:48.151860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.193:46741)
0928 08:53:48.152004 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:48.152050 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20260928 08:53:48.152160 31639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:48.151914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.195:35349)
0928 08:53:48.152068 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:48.152110 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":53,"threads_started":1}
I20260928 08:53:48.152204 31640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:48.151928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.194:50403)
0928 08:53:48.152137 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:48.152166 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":42,"threads_started":1}
I20260928 08:53:49.153136 31643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:49.152837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.195:34711)
0928 08:53:49.153012 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:49.153057 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":48,"threads_started":1}
I20260928 08:53:49.153136 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:49.152572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.193:33875)
0928 08:53:49.153012 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:49.153057 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":88,"threads_started":1}
I20260928 08:53:49.153155 31642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:49.152596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.194:38457)
0928 08:53:49.153043 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:49.153103 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":55,"threads_started":1}
I20260928 08:53:50.153889 31644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:50.153596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.193:42317)
0928 08:53:50.153790 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:50.153833 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":92,"threads_started":1}
I20260928 08:53:50.153898 31646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:50.153706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.195:56001)
0928 08:53:50.153814 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:50.153850 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":50,"threads_started":1}
I20260928 08:53:50.154168 31645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:50.153670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.194:39973)
0928 08:53:50.154092 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:50.154128 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":45,"threads_started":1}
I20260928 08:53:51.154685 31647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:51.154368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.195:46325)
0928 08:53:51.154586 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:51.154627 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":62,"threads_started":1}
I20260928 08:53:51.154713 31648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:51.154460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.193:44061)
0928 08:53:51.154625 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:51.154664 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":46,"threads_started":1}
I20260928 08:53:51.154821 31649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:51.154506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.194:57345)
0928 08:53:51.154752 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:51.154785 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":53,"threads_started":1}
I20260928 08:53:51.675733 31650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b528077fc484f4fae48815cbca7e762)
I20260928 08:53:51.675834 31650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 44651 } attrs { promote: false } } peers { permanent_uuid: "ce49edb20df84405acc0d56445850999" member_type: NON_VOTER last_known_addr { host: "127.23.167.252" port: 40727 } attrs { promote: true } }
I20260928 08:53:51.675951 31650 leader_election.cc:288] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b528077fc484f4fae48815cbca7e762 (127.23.167.254:34791)
I20260928 08:53:51.676273 31651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:51.676103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.0.0.1:59972)
0928 08:53:51.676196 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:51.676233 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":45,"threads_started":1}
I20260928 08:53:51.676658 31651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:51.676573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.0.0.1:59986)
0928 08:53:51.676615 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:51.676641 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 08:53:51.676738 30891 leader_election.cc:334] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0b528077fc484f4fae48815cbca7e762 (127.23.167.254:34791): Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
I20260928 08:53:51.676785 30891 leader_election.cc:302] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 940646d9191e48d3903cb7037d8ed33c; no voters: 0b528077fc484f4fae48815cbca7e762
I20260928 08:53:51.676842 31650 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:53:52.155460 31653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:52.155189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.193:32793)
0928 08:53:52.155372 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:52.155409 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":63,"threads_started":1}
I20260928 08:53:52.155435 31654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:52.155189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.194:54367)
0928 08:53:52.155332 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:52.155378 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":65,"threads_started":1}
I20260928 08:53:52.155733 31655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:52.155335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.195:42325)
0928 08:53:52.155662 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:52.155697 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":63,"threads_started":1}
I20260928 08:53:53.156301 31658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:53.156095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.194:36135)
0928 08:53:53.156206 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:53.156247 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":56,"threads_started":1}
I20260928 08:53:53.156301 31656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:53.155970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.193:57719)
0928 08:53:53.156208 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:53.156247 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":65,"threads_started":1}
I20260928 08:53:53.156345 31657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:53.156060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.195:57997)
0928 08:53:53.156257 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:53.156297 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":55,"threads_started":1}
I20260928 08:53:54.156960 31660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:54.156744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.195:39001)
0928 08:53:54.156865 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:54.156904 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":62,"threads_started":1}
I20260928 08:53:54.157037 31661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:54.156844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.194:34199)
0928 08:53:54.156956 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:54.156991 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":48,"threads_started":1}
I20260928 08:53:54.157111 31659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:54.156754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.193:41311)
0928 08:53:54.157012 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:54.157042 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":54,"threads_started":1}
I20260928 08:53:54.954936 31662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b528077fc484f4fae48815cbca7e762)
I20260928 08:53:54.955026 31662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 44651 } attrs { promote: false } } peers { permanent_uuid: "ce49edb20df84405acc0d56445850999" member_type: NON_VOTER last_known_addr { host: "127.23.167.252" port: 40727 } attrs { promote: true } }
I20260928 08:53:54.955168 31662 leader_election.cc:288] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b528077fc484f4fae48815cbca7e762 (127.23.167.254:34791)
I20260928 08:53:54.955462 31663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:54.955296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.0.0.1:50128)
0928 08:53:54.955388 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:54.955422 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":44,"threads_started":1}
W20260928 08:53:54.955538 30891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260928 08:53:54.955881 31663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:54.955765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.0.0.1:50136)
0928 08:53:54.955815 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:54.955855 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 08:53:54.955957 30891 leader_election.cc:334] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0b528077fc484f4fae48815cbca7e762 (127.23.167.254:34791): Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
I20260928 08:53:54.956013 30891 leader_election.cc:302] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 940646d9191e48d3903cb7037d8ed33c; no voters: 0b528077fc484f4fae48815cbca7e762
I20260928 08:53:54.956103 31662 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:53:55.157966 31666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:55.157605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.194:58347)
0928 08:53:55.157769 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:55.157818 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":88,"threads_started":1}
I20260928 08:53:55.157917 31667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:55.157605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.193:56827)
0928 08:53:55.157814 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:55.157856 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":96,"threads_started":1}
I20260928 08:53:55.158030 31665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:55.157627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.195:34993)
0928 08:53:55.157905 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:55.157936 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":70,"threads_started":1}
I20260928 08:53:56.158739 31670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:56.158541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.193:60959)
0928 08:53:56.158656 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:56.158692 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":50,"threads_started":1}
I20260928 08:53:56.158739 31668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:56.158411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.195:40927)
0928 08:53:56.158663 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:56.158691 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":115,"threads_started":1}
I20260928 08:53:56.158739 31669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:56.158484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.194:45077)
0928 08:53:56.158640 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:56.158686 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":73,"threads_started":1}
I20260928 08:53:57.159561 31671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:57.159237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.195:54177)
0928 08:53:57.159453 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:57.159503 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":81,"threads_started":1}
I20260928 08:53:57.159570 31672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:57.159244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.193:42577)
0928 08:53:57.159476 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:57.159521 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":78,"threads_started":1}
I20260928 08:53:57.159897 31673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:57.159679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.194:46363)
0928 08:53:57.159825 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:57.159856 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":50,"threads_started":1}
I20260928 08:53:58.160220 31674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:58.159980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.193:56581)
0928 08:53:58.160112 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:58.160163 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":65,"threads_started":1}
I20260928 08:53:58.160429 31675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:58.160193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.195:47489)
0928 08:53:58.160349 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:58.160382 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":61,"threads_started":1}
I20260928 08:53:58.160492 31676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:58.160241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.194:44919)
0928 08:53:58.160419 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:58.160452 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":51,"threads_started":1}
I20260928 08:53:59.161000 31677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:59.160792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.193:46611)
0928 08:53:59.160902 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:59.160945 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":62,"threads_started":1}
I20260928 08:53:59.161262 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:59.160809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.194:53925)
0928 08:53:59.161175 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:59.161213 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":53,"threads_started":1}
I20260928 08:53:59.161280 31679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:53:59.161001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.195:59715)
0928 08:53:59.161209 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 08:53:59.161240 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":84,"threads_started":1}
I20260928 08:54:00.161891 31681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:00.161631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.195:50639)
0928 08:54:00.161783 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:00.161835 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":53,"threads_started":1}
I20260928 08:54:00.161895 31680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:00.161613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.193:59833)
0928 08:54:00.161808 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:00.161845 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":73,"threads_started":1}
I20260928 08:54:00.162178 31682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:00.161859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.194:48727)
0928 08:54:00.162070 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:00.162136 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":71,"threads_started":1}
I20260928 08:54:01.162516 31683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:01.162245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.193:47763)
0928 08:54:01.162414 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:01.162459 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":65,"threads_started":1}
I20260928 08:54:01.162562 31684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:01.162257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.195:58683)
0928 08:54:01.162475 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:01.162512 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":69,"threads_started":1}
I20260928 08:54:01.162638 31685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:01.162471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.194:56383)
0928 08:54:01.162575 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:01.162602 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
I20260928 08:54:02.163196 31686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:02.162897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.195:54895)
0928 08:54:02.163081 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:02.163138 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":64,"threads_started":1}
I20260928 08:54:02.163237 31687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:02.163030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.193:42543)
0928 08:54:02.163152 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:02.163190 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":50,"threads_started":1}
I20260928 08:54:02.163298 31688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:02.163017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.194:52517)
0928 08:54:02.163224 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:02.163260 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":64,"threads_started":1}
I20260928 08:54:03.163836 31689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:03.163557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.195:56475)
0928 08:54:03.163746 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:03.163786 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":61,"threads_started":1}
I20260928 08:54:03.163836 31690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:03.163614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.194:40413)
0928 08:54:03.163756 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:03.163786 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":53,"threads_started":1}
I20260928 08:54:03.163853 31691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:03.163685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.193:57599)
0928 08:54:03.163773 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:03.163809 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71,"thread_start_us":41,"threads_started":1}
I20260928 08:54:03.722397 31692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b528077fc484f4fae48815cbca7e762)
I20260928 08:54:03.722503 31692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0b528077fc484f4fae48815cbca7e762" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34791 } } peers { permanent_uuid: "940646d9191e48d3903cb7037d8ed33c" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 44651 } attrs { promote: false } } peers { permanent_uuid: "ce49edb20df84405acc0d56445850999" member_type: NON_VOTER last_known_addr { host: "127.23.167.252" port: 40727 } attrs { promote: true } }
I20260928 08:54:03.722651 31692 leader_election.cc:288] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b528077fc484f4fae48815cbca7e762 (127.23.167.254:34791)
I20260928 08:54:03.723003 31693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:03.722816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.0.0.1:51970)
0928 08:54:03.722912 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:03.722956 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":43,"threads_started":1}
W20260928 08:54:03.723088 30891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
I20260928 08:54:03.723421 31693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:03.723329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.0.0.1:51978)
0928 08:54:03.723375 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:03.723402 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 08:54:03.723515 30891 leader_election.cc:334] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0b528077fc484f4fae48815cbca7e762 (127.23.167.254:34791): Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
I20260928 08:54:03.723567 30891 leader_election.cc:302] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 940646d9191e48d3903cb7037d8ed33c; no voters: 0b528077fc484f4fae48815cbca7e762
I20260928 08:54:03.723644 31692 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 940646d9191e48d3903cb7037d8ed33c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:54:04.164443 31695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:04.164219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.194:44195)
0928 08:54:04.164345 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:04.164387 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":60,"threads_started":1}
I20260928 08:54:04.164451 31696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:04.164213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.195:59507)
0928 08:54:04.164358 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:04.164399 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":74,"threads_started":1}
I20260928 08:54:04.164729 31697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:04.164499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.193:35521)
0928 08:54:04.164669 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:04.164697 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":59,"threads_started":1}
I20260928 08:54:05.165170 31698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:05.164903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.194:43551)
0928 08:54:05.165031 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:05.165112 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260928 08:54:05.165220 31699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:05.164990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.195:52975)
0928 08:54:05.165132 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:05.165170 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":46,"threads_started":1}
I20260928 08:54:05.165444 31700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:05.165226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.193:46365)
0928 08:54:05.165372 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:05.165406 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":48,"threads_started":1}
I20260928 08:54:06.165920 31701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:06.165576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.194:48461)
0928 08:54:06.165808 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:06.165861 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":98,"threads_started":1}
I20260928 08:54:06.165920 31702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:06.165669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.195:58863)
0928 08:54:06.165833 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:06.165870 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":55,"threads_started":1}
I20260928 08:54:06.166183 31703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:06.165886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.193:55143)
0928 08:54:06.166105 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:06.166140 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":60,"threads_started":1}
I20260928 08:54:07.166800 31704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:07.166481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.195:49473)
0928 08:54:07.166665 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:07.166739 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":65,"threads_started":1}
I20260928 08:54:07.166842 31705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:07.166514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.194:42747)
0928 08:54:07.166731 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:07.166769 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":41,"threads_started":1}
I20260928 08:54:07.166819 31706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:07.166561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.193:48347)
0928 08:54:07.166731 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:07.166769 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":58,"threads_started":1}
I20260928 08:54:08.167529 31708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:08.167264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.193:37839)
0928 08:54:08.167438 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:08.167483 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":55,"threads_started":1}
I20260928 08:54:08.167577 31707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:08.167220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.194:50909)
0928 08:54:08.167442 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:08.167494 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":79,"threads_started":1}
I20260928 08:54:08.167634 31709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:08.167439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.195:45567)
0928 08:54:08.167567 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:08.167596 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":44,"threads_started":1}
I20260928 08:54:09.168334 31710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:09.168027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.193:48793)
0928 08:54:09.168198 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:09.168238 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":62,"threads_started":1}
I20260928 08:54:09.168344 31712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:09.168111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.194:33407)
0928 08:54:09.168271 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:09.168304 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":51,"threads_started":1}
I20260928 08:54:09.168285 31711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:09.168027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:34791 (local address 127.23.167.195:41249)
0928 08:54:09.168204 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:09.168237 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":66,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.23.167.254:34791,127.23.167.253:44651,127.23.167.252:40727, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.23.167.254:34791,127.23.167.253:44651,127.23.167.252:40727, num_attempts: 1)
I20260928 08:54:09.340585 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 31053
I20260928 08:54:09.345515 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 31279
I20260928 08:54:09.349603 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 31415
I20260928 08:54:09.353514 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 30771
I20260928 08:54:09.357173 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 30960
2026-09-28T08:54:09Z chronyd exiting
I20260928 08:54:09.372079 24223 test_util.cc:175] -----------------------------------------------
I20260928 08:54:09.372133 24223 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585589999956-24223-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (36358 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:54:09 dist-test-slave-pbjp krb5kdc[31715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:54:09 dist-test-slave-pbjp krb5kdc[31715](info): set up 2 sockets
Sep 28 08:54:09 dist-test-slave-pbjp krb5kdc[31715](info): commencing operation
krb5kdc: starting...
W20260928 08:54:10.313176 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:54:10 dist-test-slave-pbjp krb5kdc[31715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:54:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:54:10Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.167.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.167.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.167.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.254.keytab.
Entry for principal oryx/127.23.167.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.254.keytab.
I20260928 08:54:10.423501 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:42661
--webserver_interface=127.23.167.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.254.keytab
--principal=oryx/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:33653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:42661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:54:10.502106 31728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:54:10.502313 31728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:54:10.502343 31728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:54:10.503489 31728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:54:10.503527 31728 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 08:54:10.503544 31728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:54:10.503561 31728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:54:10.503575 31728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:54:10.503590 31728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:54:10.505230 31728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:33653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:42661
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.254.keytab
--principal=oryx/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:42661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:54:10.505482 31728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:54:10.505682 31728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:54:10.510301 31733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:10.510314 31736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:10.510406 31728 server_base.cc:1034] running on GCE node
W20260928 08:54:10.510293 31734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:10.510886 31728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:54:10.511168 31728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:54:10.512306 31728 hybrid_clock.cc:648] HybridClock initialized: now 1790585650512293 us; error 28 us; skew 500 ppm
Sep 28 08:54:10 dist-test-slave-pbjp krb5kdc[31715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585650, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:54:10.514745 31728 init.cc:377] Logged in from keytab as oryx/127.23.167.254@KRBTEST.COM (short username oryx)
I20260928 08:54:10.515159 31728 webserver.cc:460] Webserver started at http://127.23.167.254:41367/ using document root <none> and password file <none>
I20260928 08:54:10.515367 31728 fs_manager.cc:359] Metadata directory not provided
I20260928 08:54:10.515415 31728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:54:10.515576 31728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:54:10.516621 31728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "dc1b95ca7c2d48d5ae1c2c1064bcb2b1"
format_stamp: "Formatted at 2026-09-28 08:54:10 on dist-test-slave-pbjp"
server_key: "72f3aee043119251cb93687d54f00c3e"
server_key_iv: "3bdef1fb39955b6e037ba0b15733fe70"
server_key_version: "encryptionkey@0"
I20260928 08:54:10.516948 31728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "dc1b95ca7c2d48d5ae1c2c1064bcb2b1"
format_stamp: "Formatted at 2026-09-28 08:54:10 on dist-test-slave-pbjp"
server_key: "72f3aee043119251cb93687d54f00c3e"
server_key_iv: "3bdef1fb39955b6e037ba0b15733fe70"
server_key_version: "encryptionkey@0"
I20260928 08:54:10.518224 31728 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:54:10.518878 31743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:10.519019 31728 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:54:10.519097 31728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "dc1b95ca7c2d48d5ae1c2c1064bcb2b1"
format_stamp: "Formatted at 2026-09-28 08:54:10 on dist-test-slave-pbjp"
server_key: "72f3aee043119251cb93687d54f00c3e"
server_key_iv: "3bdef1fb39955b6e037ba0b15733fe70"
server_key_version: "encryptionkey@0"
I20260928 08:54:10.519157 31728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:54:10.527208 31728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:54:10.529368 31728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:54:10.529551 31728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:54:10.533951 31728 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:42661
I20260928 08:54:10.534001 31805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:42661 every 8 connection(s)
I20260928 08:54:10.534330 31728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:54:10.535039 31806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:54:10.537537 31806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc1b95ca7c2d48d5ae1c2c1064bcb2b1: Bootstrap starting.
I20260928 08:54:10.537830 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 31728
I20260928 08:54:10.537915 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:54:10.538069 24223 external_mini_cluster.cc:1428] Setting key 58d984ca693bb87be1b942577eda2614
I20260928 08:54:10.538197 31806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc1b95ca7c2d48d5ae1c2c1064bcb2b1: Neither blocks nor log segments found. Creating new log.
I20260928 08:54:10.538475 31806 log.cc:824] T 00000000000000000000000000000000 P dc1b95ca7c2d48d5ae1c2c1064bcb2b1: Log is configured to *not* fsync() on all Append() calls
I20260928 08:54:10.539244 31806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc1b95ca7c2d48d5ae1c2c1064bcb2b1: No bootstrap required, opened a new log
I20260928 08:54:10.541623 31806 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc1b95ca7c2d48d5ae1c2c1064bcb2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1b95ca7c2d48d5ae1c2c1064bcb2b1" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42661 } }
I20260928 08:54:10.541829 31806 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc1b95ca7c2d48d5ae1c2c1064bcb2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:54:10.541878 31806 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc1b95ca7c2d48d5ae1c2c1064bcb2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc1b95ca7c2d48d5ae1c2c1064bcb2b1, State: Initialized, Role: FOLLOWER
I20260928 08:54:10.542025 31806 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc1b95ca7c2d48d5ae1c2c1064bcb2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1b95ca7c2d48d5ae1c2c1064bcb2b1" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42661 } }
I20260928 08:54:10.542094 31806 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc1b95ca7c2d48d5ae1c2c1064bcb2b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:54:10.542124 31806 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc1b95ca7c2d48d5ae1c2c1064bcb2b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:54:10.542171 31806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc1b95ca7c2d48d5ae1c2c1064bcb2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:54:10.542819 31806 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc1b95ca7c2d48d5ae1c2c1064bcb2b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1b95ca7c2d48d5ae1c2c1064bcb2b1" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42661 } }
I20260928 08:54:10.542937 31806 leader_election.cc:302] T 00000000000000000000000000000000 P dc1b95ca7c2d48d5ae1c2c1064bcb2b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc1b95ca7c2d48d5ae1c2c1064bcb2b1; no voters:
I20260928 08:54:10.543107 31806 leader_election.cc:288] T 00000000000000000000000000000000 P dc1b95ca7c2d48d5ae1c2c1064bcb2b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:54:10.543157 31811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc1b95ca7c2d48d5ae1c2c1064bcb2b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:54:10.543358 31806 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc1b95ca7c2d48d5ae1c2c1064bcb2b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:54:10.543368 31811 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc1b95ca7c2d48d5ae1c2c1064bcb2b1 [term 1 LEADER]: Becoming Leader. State: Replica: dc1b95ca7c2d48d5ae1c2c1064bcb2b1, State: Running, Role: LEADER
I20260928 08:54:10.543828 31811 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc1b95ca7c2d48d5ae1c2c1064bcb2b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1b95ca7c2d48d5ae1c2c1064bcb2b1" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42661 } }
I20260928 08:54:10.545503 31813 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc1b95ca7c2d48d5ae1c2c1064bcb2b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc1b95ca7c2d48d5ae1c2c1064bcb2b1. Latest consensus state: current_term: 1 leader_uuid: "dc1b95ca7c2d48d5ae1c2c1064bcb2b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1b95ca7c2d48d5ae1c2c1064bcb2b1" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42661 } } }
Sep 28 08:54:10 dist-test-slave-pbjp krb5kdc[31715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM
I20260928 08:54:10.545605 31813 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc1b95ca7c2d48d5ae1c2c1064bcb2b1 [sys.catalog]: This master's current role is: LEADER
I20260928 08:54:10.545787 31812 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc1b95ca7c2d48d5ae1c2c1064bcb2b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc1b95ca7c2d48d5ae1c2c1064bcb2b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1b95ca7c2d48d5ae1c2c1064bcb2b1" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42661 } } }
I20260928 08:54:10.545859 31812 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc1b95ca7c2d48d5ae1c2c1064bcb2b1 [sys.catalog]: This master's current role is: LEADER
W20260928 08:54:10.546315 31825 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc1b95ca7c2d48d5ae1c2c1064bcb2b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:54:10.546371 31825 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:54:10.546710 31826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:54:10.546873 31826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:54:10.548456 31826 catalog_manager.cc:1347] Generated new cluster ID: 4703edcda83041df8d0b26859717a20f
I20260928 08:54:10.548509 31826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:54:10.549606 31810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:10.538640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35390 (local address 127.23.167.254:42661)
0928 08:54:10.538812 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:10.538821 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:10.538855 (+ 34us) server_negotiation.cc:407] Connection header received
0928 08:54:10.539447 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:10.539457 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:10.539526 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:10.539688 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:10.540288 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:10.541754 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:10.542697 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:10.543328 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:10.543494 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:10.546200 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:54:10.546205 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:54:10.546206 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:54:10.546219 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:54:10.547967 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:10.548359 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:10.548361 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:10.548362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:10.548465 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:10.548722 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:10.548723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:10.548723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:10.548970 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:54:10.549126 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:10.549245 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 08:54:10.549393 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
I20260928 08:54:10.563593 31826 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:54:10.564145 31826 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:54:10.569391 31826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc1b95ca7c2d48d5ae1c2c1064bcb2b1: Generated new TSK 0
I20260928 08:54:10.569576 31826 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.167.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.167.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.167.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.193.keytab.
Entry for principal oryx/127.23.167.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.193.keytab.
I20260928 08:54:10.599172 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:0
--local_ip_for_outbound_sockets=127.23.167.193
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:42661
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.193.keytab
--principal=oryx/127.23.167.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:33653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:54:10.680832 31832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:54:10.681037 31832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:54:10.681087 31832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:54:10.682235 31832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:54:10.682271 31832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:54:10.682297 31832 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:54:10.682319 31832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:54:10.683802 31832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:33653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.193.keytab
--principal=oryx/127.23.167.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.167.254:42661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:54:10.684023 31832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:54:10.684229 31832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:54:10.688841 31837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:10.688911 31838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:10.688841 31840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:10.689103 31832 server_base.cc:1034] running on GCE node
I20260928 08:54:10.689314 31832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:54:10.689540 31832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:54:10.690711 31832 hybrid_clock.cc:648] HybridClock initialized: now 1790585650690694 us; error 32 us; skew 500 ppm
Sep 28 08:54:10 dist-test-slave-pbjp krb5kdc[31715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585650, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:54:10.693248 31832 init.cc:377] Logged in from keytab as oryx/127.23.167.193@KRBTEST.COM (short username oryx)
I20260928 08:54:10.693698 31832 webserver.cc:460] Webserver started at http://127.23.167.193:33801/ using document root <none> and password file <none>
I20260928 08:54:10.693888 31832 fs_manager.cc:359] Metadata directory not provided
I20260928 08:54:10.693935 31832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:54:10.694056 31832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:54:10.695077 31832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-0/data/instance:
uuid: "c7fa0d4b654942aca29b1488c9dacb1b"
format_stamp: "Formatted at 2026-09-28 08:54:10 on dist-test-slave-pbjp"
server_key: "c0045fb9cca0511ee293333981304d93"
server_key_iv: "18c030726a04a2be22bab2917af858fe"
server_key_version: "encryptionkey@0"
I20260928 08:54:10.695426 31832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance:
uuid: "c7fa0d4b654942aca29b1488c9dacb1b"
format_stamp: "Formatted at 2026-09-28 08:54:10 on dist-test-slave-pbjp"
server_key: "c0045fb9cca0511ee293333981304d93"
server_key_iv: "18c030726a04a2be22bab2917af858fe"
server_key_version: "encryptionkey@0"
I20260928 08:54:10.696727 31832 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 08:54:10.697397 31847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:10.697544 31832 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:10.697628 31832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "c7fa0d4b654942aca29b1488c9dacb1b"
format_stamp: "Formatted at 2026-09-28 08:54:10 on dist-test-slave-pbjp"
server_key: "c0045fb9cca0511ee293333981304d93"
server_key_iv: "18c030726a04a2be22bab2917af858fe"
server_key_version: "encryptionkey@0"
I20260928 08:54:10.697687 31832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:54:10.704640 31832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:54:10.706498 31832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:54:10.706629 31832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:54:10.707161 31832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:54:10.707209 31832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:10.707237 31832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:54:10.707259 31832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:10.713277 31832 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:46433
I20260928 08:54:10.713336 31962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:46433 every 8 connection(s)
Sep 28 08:54:10 dist-test-slave-pbjp krb5kdc[31715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585650, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM
I20260928 08:54:10.713682 31832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
I20260928 08:54:10.714265 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 31832
I20260928 08:54:10.714383 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance
I20260928 08:54:10.714586 24223 external_mini_cluster.cc:1428] Setting key ea2e7593e68a7b34c8b91913ab1a67b9
I20260928 08:54:10.716864 31810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:10.707535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:38023 (local address 127.23.167.254:42661)
0928 08:54:10.707583 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:10.707585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:10.707783 (+ 198us) server_negotiation.cc:407] Connection header received
0928 08:54:10.708291 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:10.708293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:10.708305 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:10.708356 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:10.709180 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:10.709741 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:10.711194 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:10.711502 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:10.711571 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:10.714111 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:54:10.714114 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:54:10.714114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:54:10.714123 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:54:10.715600 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:10.715965 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:10.715966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:10.715966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:10.716023 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:10.716214 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:10.716215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:10.716215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:10.716335 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:54:10.716397 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:10.716678 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 08:54:10.716773 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 08:54:10.717015 31856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:10.707536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42661 (local address 127.23.167.193:38023)
0928 08:54:10.707673 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:10.707709 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:10.708191 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:10.708439 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:10.708443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:10.708787 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:10.709095 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:10.709103 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:10.709830 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:10.709832 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:10.711110 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:10.711114 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:10.711579 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:10.711580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:10.711709 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:10.712252 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:10.712263 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:10.714030 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:54:10.715676 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:54:10.715681 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:54:10.715684 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:54:10.715901 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:54:10.716076 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:54:10.716077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:54:10.716077 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:54:10.716167 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:54:10.716388 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:54:10.716391 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:54:10.716544 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 08:54:10.716642 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 08:54:10.716802 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":76,"threads_started":1}
I20260928 08:54:10.717664 31963 heartbeater.cc:347] Connected to a master server at 127.23.167.254:42661
I20260928 08:54:10.717747 31963 heartbeater.cc:464] Registering TS with master...
I20260928 08:54:10.717921 31963 heartbeater.cc:511] Master 127.23.167.254:42661 requested a full tablet report, sending...
I20260928 08:54:10.718456 31759 ts_manager.cc:194] Registered new tserver with Master: c7fa0d4b654942aca29b1488c9dacb1b (127.23.167.193:46433)
I20260928 08:54:10.719312 31759 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.167.193@KRBTEST.COM'} at 127.23.167.193:38023
WARNING: no policy specified for oryx/127.23.167.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.167.194@KRBTEST.COM" created.
I20260928 08:54:10.728860 31970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:10.719664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42661 (local address 127.23.167.193:53379)
0928 08:54:10.723570 (+ 3906us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:10.723582 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:10.723643 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:10.723756 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:10.723757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:10.723912 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:10.723966 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:10.723967 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:10.724638 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:10.724638 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:10.725298 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:10.725300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:10.725751 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:10.725752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:10.725830 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:10.726294 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:10.726298 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:10.726585 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:54:10.727952 (+ 1367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:54:10.727953 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:54:10.727954 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:54:10.728113 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:54:10.728271 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:54:10.728272 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:54:10.728272 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:54:10.728329 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:54:10.728541 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:54:10.728542 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:54:10.728589 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 08:54:10.728648 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 08:54:10.728754 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3876,"thread_start_us":63,"threads_started":1}
I20260928 08:54:10.728856 31810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:10.719715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:53379 (local address 127.23.167.254:42661)
0928 08:54:10.719754 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:10.719756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:10.723643 (+ 3887us) server_negotiation.cc:407] Connection header received
0928 08:54:10.723692 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:10.723693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:10.723704 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:10.723751 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:10.724011 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:10.724560 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:10.725418 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:10.725622 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:10.725685 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:10.726668 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:54:10.726669 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:54:10.726670 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:54:10.726684 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:54:10.727878 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:10.728171 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:10.728172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:10.728172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:10.728221 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:10.728377 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:10.728378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:10.728378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:10.728483 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:54:10.728542 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:10.728680 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 08:54:10.728752 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.167.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.194.keytab.
Entry for principal oryx/127.23.167.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.194.keytab.
I20260928 08:54:10.740404 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:0
--local_ip_for_outbound_sockets=127.23.167.194
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:42661
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.194.keytab
--principal=oryx/127.23.167.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:33653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:54:10.819974 31972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:54:10.820179 31972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:54:10.820204 31972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:54:10.821381 31972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:54:10.821427 31972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:54:10.821449 31972 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:54:10.821480 31972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:54:10.822989 31972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:33653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.194.keytab
--principal=oryx/127.23.167.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.167.254:42661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:54:10.823204 31972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:54:10.823410 31972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:54:10.828327 31977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:10.828320 31980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:10.828320 31978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:10.828334 31972 server_base.cc:1034] running on GCE node
I20260928 08:54:10.828713 31972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:54:10.828953 31972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:54:10.830086 31972 hybrid_clock.cc:648] HybridClock initialized: now 1790585650830075 us; error 33 us; skew 500 ppm
Sep 28 08:54:10 dist-test-slave-pbjp krb5kdc[31715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585650, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:54:10.832506 31972 init.cc:377] Logged in from keytab as oryx/127.23.167.194@KRBTEST.COM (short username oryx)
I20260928 08:54:10.832928 31972 webserver.cc:460] Webserver started at http://127.23.167.194:46449/ using document root <none> and password file <none>
I20260928 08:54:10.833176 31972 fs_manager.cc:359] Metadata directory not provided
I20260928 08:54:10.833227 31972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:54:10.833348 31972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:54:10.834384 31972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-1/data/instance:
uuid: "b4a99b7bc9f54355980d72922c77af06"
format_stamp: "Formatted at 2026-09-28 08:54:10 on dist-test-slave-pbjp"
server_key: "0913880c7ba4e7d9d98f1dc1bf0a9d89"
server_key_iv: "3257c77538ef5111afb32709a418147c"
server_key_version: "encryptionkey@0"
I20260928 08:54:10.834719 31972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance:
uuid: "b4a99b7bc9f54355980d72922c77af06"
format_stamp: "Formatted at 2026-09-28 08:54:10 on dist-test-slave-pbjp"
server_key: "0913880c7ba4e7d9d98f1dc1bf0a9d89"
server_key_iv: "3257c77538ef5111afb32709a418147c"
server_key_version: "encryptionkey@0"
I20260928 08:54:10.835937 31972 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:54:10.836810 31987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:10.837378 31972 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:54:10.837464 31972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "b4a99b7bc9f54355980d72922c77af06"
format_stamp: "Formatted at 2026-09-28 08:54:10 on dist-test-slave-pbjp"
server_key: "0913880c7ba4e7d9d98f1dc1bf0a9d89"
server_key_iv: "3257c77538ef5111afb32709a418147c"
server_key_version: "encryptionkey@0"
I20260928 08:54:10.837533 31972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:54:10.852595 31972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:54:10.854962 31972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:54:10.855109 31972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:54:10.855635 31972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:54:10.855677 31972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:10.855707 31972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:54:10.855723 31972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:10.862496 31972 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:33563
I20260928 08:54:10.862586 32102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:33563 every 8 connection(s)
Sep 28 08:54:10 dist-test-slave-pbjp krb5kdc[31715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585650, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM
I20260928 08:54:10.862895 31972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
I20260928 08:54:10.865197 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 31972
I20260928 08:54:10.865298 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance
I20260928 08:54:10.865504 24223 external_mini_cluster.cc:1428] Setting key 2339a226518ecdf3f3a537eb9520b7a3
I20260928 08:54:10.865703 31810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:10.856078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:44721 (local address 127.23.167.254:42661)
0928 08:54:10.856143 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:10.856145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:10.856358 (+ 213us) server_negotiation.cc:407] Connection header received
0928 08:54:10.857047 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:10.857048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:10.857057 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:10.857125 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:10.858000 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:10.858760 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:10.860290 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:10.860592 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:10.860656 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:10.863221 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:54:10.863222 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:54:10.863223 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:54:10.863231 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:54:10.864457 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:10.864799 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:10.864800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:10.864800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:10.864845 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:10.865022 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:10.865022 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:10.865022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:10.865163 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:54:10.865225 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:10.865531 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 08:54:10.865615 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 08:54:10.865857 31996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:10.856050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42661 (local address 127.23.167.194:44721)
0928 08:54:10.856215 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:10.856266 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:10.856938 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:10.857218 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:10.857222 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:10.857600 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:10.857898 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:10.857906 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:10.858861 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:10.858862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:10.860196 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:10.860200 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:10.860673 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:10.860674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:10.860826 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:10.861437 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:10.861448 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:10.863112 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:54:10.864528 (+ 1416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:54:10.864533 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:54:10.864534 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:54:10.864741 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:54:10.864883 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:54:10.864883 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:54:10.864884 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:54:10.864968 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:54:10.865227 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:54:10.865235 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:54:10.865391 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 08:54:10.865497 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 08:54:10.865651 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260928 08:54:10.866422 32103 heartbeater.cc:347] Connected to a master server at 127.23.167.254:42661
I20260928 08:54:10.866511 32103 heartbeater.cc:464] Registering TS with master...
I20260928 08:54:10.866689 32103 heartbeater.cc:511] Master 127.23.167.254:42661 requested a full tablet report, sending...
I20260928 08:54:10.867501 31760 ts_manager.cc:194] Registered new tserver with Master: b4a99b7bc9f54355980d72922c77af06 (127.23.167.194:33563)
I20260928 08:54:10.868001 31760 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.167.194@KRBTEST.COM'} at 127.23.167.194:44721
I20260928 08:54:10.873518 31810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:10.868125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:52503 (local address 127.23.167.254:42661)
0928 08:54:10.868171 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:10.868172 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:10.868180 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:54:10.868200 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:10.868201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:10.868210 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:10.868263 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:10.868664 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:10.869245 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:10.870164 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:10.870342 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:10.870389 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:10.871306 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:54:10.871307 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:54:10.871307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:54:10.871313 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:54:10.872452 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:10.872811 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:10.872811 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:10.872811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:10.872853 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:10.873019 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:10.873019 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:10.873020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:10.873171 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:54:10.873221 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:10.873402 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 08:54:10.873457 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 08:54:10.873651 32109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:10.867405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42661 (local address 127.23.167.194:52503)
0928 08:54:10.867523 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:10.867534 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:10.867586 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:10.868289 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:10.868290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:10.868498 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:10.868574 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:10.868576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:10.869345 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:10.869346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:10.870091 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:10.870095 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:10.870406 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:10.870407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:10.870485 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:10.870858 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:10.870864 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:10.871226 (+ 362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:54:10.872536 (+ 1310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:54:10.872537 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:54:10.872537 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:54:10.872742 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:54:10.872889 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:54:10.872890 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:54:10.872890 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:54:10.872964 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:54:10.873248 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:54:10.873249 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:54:10.873300 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 08:54:10.873379 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 08:54:10.873485 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for oryx/127.23.167.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.167.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.167.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.195.keytab.
Entry for principal oryx/127.23.167.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.195.keytab.
I20260928 08:54:10.891593 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:0
--local_ip_for_outbound_sockets=127.23.167.195
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:42661
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.195.keytab
--principal=oryx/127.23.167.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:33653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:54:10.971397 32112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:54:10.971598 32112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:54:10.971621 32112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:54:10.972724 32112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:54:10.972759 32112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:54:10.972776 32112 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:54:10.972792 32112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:54:10.974295 32112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:33653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.195.keytab
--principal=oryx/127.23.167.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.167.254:42661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:54:10.974493 32112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:54:10.974709 32112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:54:10.979447 32117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:10.979485 32118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:10.979462 32112 server_base.cc:1034] running on GCE node
W20260928 08:54:10.979447 32120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:10.979853 32112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:54:10.980100 32112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:54:10.981282 32112 hybrid_clock.cc:648] HybridClock initialized: now 1790585650981271 us; error 29 us; skew 500 ppm
Sep 28 08:54:10 dist-test-slave-pbjp krb5kdc[31715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585650, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:54:10.983721 32112 init.cc:377] Logged in from keytab as oryx/127.23.167.195@KRBTEST.COM (short username oryx)
I20260928 08:54:10.984148 32112 webserver.cc:460] Webserver started at http://127.23.167.195:35389/ using document root <none> and password file <none>
I20260928 08:54:10.984360 32112 fs_manager.cc:359] Metadata directory not provided
I20260928 08:54:10.984438 32112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:54:10.984566 32112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:54:10.985680 32112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-2/data/instance:
uuid: "80ec10c0466246e5a4fb75ce61877a1f"
format_stamp: "Formatted at 2026-09-28 08:54:10 on dist-test-slave-pbjp"
server_key: "bad4f369c6ffa1b15f8ef513c1c0b0b2"
server_key_iv: "83a196669fb0e934d008495e0807e99c"
server_key_version: "encryptionkey@0"
I20260928 08:54:10.985987 32112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance:
uuid: "80ec10c0466246e5a4fb75ce61877a1f"
format_stamp: "Formatted at 2026-09-28 08:54:10 on dist-test-slave-pbjp"
server_key: "bad4f369c6ffa1b15f8ef513c1c0b0b2"
server_key_iv: "83a196669fb0e934d008495e0807e99c"
server_key_version: "encryptionkey@0"
I20260928 08:54:10.987157 32112 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 08:54:10.987821 32127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:10.987954 32112 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:54:10.988034 32112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "80ec10c0466246e5a4fb75ce61877a1f"
format_stamp: "Formatted at 2026-09-28 08:54:10 on dist-test-slave-pbjp"
server_key: "bad4f369c6ffa1b15f8ef513c1c0b0b2"
server_key_iv: "83a196669fb0e934d008495e0807e99c"
server_key_version: "encryptionkey@0"
I20260928 08:54:10.988090 32112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:54:11.006348 32112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:54:11.008656 32112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:54:11.008807 32112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:54:11.009419 32112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:54:11.009465 32112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:11.009496 32112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:54:11.009511 32112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:54:11 dist-test-slave-pbjp krb5kdc[31715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585650, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM
I20260928 08:54:11.016110 32112 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:42447
I20260928 08:54:11.016191 32242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:42447 every 8 connection(s)
I20260928 08:54:11.016479 32112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
I20260928 08:54:11.018728 31810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.009780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:46503 (local address 127.23.167.254:42661)
0928 08:54:11.009835 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:11.009838 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:11.010048 (+ 210us) server_negotiation.cc:407] Connection header received
0928 08:54:11.010537 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:11.010539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:11.010550 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:11.010633 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:11.011240 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.012010 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.013092 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.013308 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.013366 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.015809 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:54:11.015811 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:54:11.015811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:54:11.015819 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:54:11.017318 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:11.017702 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:11.017703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:11.017704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:11.017762 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:11.018045 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:11.018046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:11.018047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:11.018190 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:54:11.018252 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:11.018524 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 08:54:11.018613 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 08:54:11.018901 32136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.009757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42661 (local address 127.23.167.195:46503)
0928 08:54:11.009918 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:11.009961 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:11.010446 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:11.010706 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:11.010711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:11.010973 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:11.011169 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.011175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.012105 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.012106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.012985 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.012988 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.013375 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.013376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.013512 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.013991 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:11.014000 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:11.015718 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:54:11.017388 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:54:11.017394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:54:11.017396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:54:11.017643 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:54:11.017814 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:54:11.017815 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:54:11.017815 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:54:11.017913 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:54:11.018255 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:54:11.018262 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:54:11.018400 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 08:54:11.018502 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 08:54:11.018665 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20260928 08:54:11.019372 32243 heartbeater.cc:347] Connected to a master server at 127.23.167.254:42661
I20260928 08:54:11.019454 32243 heartbeater.cc:464] Registering TS with master...
I20260928 08:54:11.019668 32243 heartbeater.cc:511] Master 127.23.167.254:42661 requested a full tablet report, sending...
I20260928 08:54:11.020040 31759 ts_manager.cc:194] Registered new tserver with Master: 80ec10c0466246e5a4fb75ce61877a1f (127.23.167.195:42447)
I20260928 08:54:11.020476 31759 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.167.195@KRBTEST.COM'} at 127.23.167.195:46503
I20260928 08:54:11.025601 31810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.020419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:59845 (local address 127.23.167.254:42661)
0928 08:54:11.020455 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:11.020456 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:11.020524 (+ 68us) server_negotiation.cc:407] Connection header received
0928 08:54:11.020575 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:11.020576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:11.020584 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:11.020638 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:11.020909 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.021479 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.022137 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.022322 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.022383 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.023260 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:54:11.023262 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:54:11.023262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:54:11.023269 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:54:11.024593 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:11.024886 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:11.024886 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:11.024887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:11.024941 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:11.025131 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:11.025132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:11.025132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:11.025237 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:54:11.025291 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:11.025429 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 08:54:11.025506 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 08:54:11.025612 32249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.020360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42661 (local address 127.23.167.195:59845)
0928 08:54:11.020471 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:11.020481 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:11.020531 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:11.020642 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:11.020643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:11.020785 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:11.020839 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.020840 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.021572 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.021573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.022075 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.022077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.022384 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.022384 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.022432 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.022841 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:11.022845 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:11.023135 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:54:11.024666 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:54:11.024667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:54:11.024667 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:54:11.024826 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:54:11.024996 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:54:11.024997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:54:11.024998 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:54:11.025053 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:54:11.025291 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:54:11.025291 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:54:11.025337 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 08:54:11.025399 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 08:54:11.025503 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":51,"threads_started":1}
I20260928 08:54:11.026587 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 32112
I20260928 08:54:11.026707 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance
I20260928 08:54:11.026912 24223 external_mini_cluster.cc:1428] Setting key 90fed943ecd58b9b75a4df39ebea9a98
I20260928 08:54:11.027745 24223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 08:54:11 dist-test-slave-pbjp krb5kdc[31715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585651, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 08:54:11 dist-test-slave-pbjp krb5kdc[31715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:11 dist-test-slave-pbjp krb5kdc[31715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 08:54:11.047768 31810 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:11.042388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35394 (local address 127.23.167.254:42661)
0928 08:54:11.042449 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:11.042451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:11.042461 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:54:11.042488 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:11.042488 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:11.042497 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:11.042546 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:11.043178 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.043803 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.044599 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.044830 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.044891 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.047692 (+ 2801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35394: BlockingRecv error: recv got EOF from 127.0.0.1:35394 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 28 08:54:11 dist-test-slave-pbjp krb5kdc[31715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585651, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM
I20260928 08:54:11.054900 31810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.048486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35408 (local address 127.23.167.254:42661)
0928 08:54:11.048531 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:11.048532 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:11.048541 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:54:11.048604 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:11.048604 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:11.048614 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:11.048657 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:11.049111 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.049654 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.050336 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.050521 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.050575 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.052603 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:54:11.052604 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:54:11.052605 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:54:11.052612 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:54:11.053865 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:11.054200 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:11.054201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:11.054202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:11.054256 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:11.054420 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:11.054421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:11.054421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:11.054536 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:54:11.054623 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:11.054725 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 08:54:11.054803 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 08:54:11.056524 31759 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:54:11.058246 31759 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:54:11.064360 32262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.060254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:46433 (local address 127.0.0.1:37296)
0928 08:54:11.060620 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:11.060628 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:11.060663 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:11.060951 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:11.060952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:11.060954 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:54:11.061007 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.061009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.062334 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.062335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.063184 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.063187 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.064063 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.064064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.064128 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.064140 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 08:54:11.064209 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 08:54:11.064266 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":9,"thread_start_us":52,"threads_started":1}
I20260928 08:54:11.064474 32266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.060543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37296 (local address 127.23.167.193:46433)
0928 08:54:11.060782 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:11.060786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:11.060793 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:54:11.060825 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:11.060828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:11.060843 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:11.060944 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:54:11.061213 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.062252 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.063271 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.063979 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.064073 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.064135 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:11.064253 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 08:54:11.064297 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":91,"threads_started":1}
I20260928 08:54:11.064607 32264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.060250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:42447 (local address 127.0.0.1:46584)
0928 08:54:11.060422 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:11.060464 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:11.060530 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:11.060872 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:11.060875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:11.060878 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:54:11.060958 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.060960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.062293 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.062294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.063401 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.063402 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.064282 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.064283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.064356 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.064361 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 08:54:11.064392 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 08:54:11.064435 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":22,"thread_start_us":64,"threads_started":1}
I20260928 08:54:11.065032 32265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.060526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46584 (local address 127.23.167.195:42447)
0928 08:54:11.060697 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:11.060702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:11.060712 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:54:11.060749 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:11.060751 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:11.060767 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:11.060844 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:54:11.061045 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.062209 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.063484 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.064194 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.064786 (+ 592us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.064871 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:11.064899 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 08:54:11.064941 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":100,"threads_started":1}
I20260928 08:54:11.065169 31897 tablet_service.cc:1467] Processing CreateTablet for tablet 564addc519924983a4f1c3f32637a2f5 (DEFAULT_TABLE table=test-table [id=2a4e947297214cac888c9ea517dbd150]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:54:11.065606 31897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 564addc519924983a4f1c3f32637a2f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:54:11.065802 32263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.060254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:33563 (local address 127.0.0.1:56940)
0928 08:54:11.060437 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:11.060467 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:11.060530 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:11.061101 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:11.061102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:11.061104 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:54:11.061145 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.061147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.062092 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.062093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.063399 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.063401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.065570 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.065571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.065638 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.065641 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:54:11.065689 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 08:54:11.065735 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":2,"thread_start_us":60,"threads_started":1}
I20260928 08:54:11.065882 32267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.060730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56940 (local address 127.23.167.194:33563)
0928 08:54:11.060921 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:11.060924 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:11.060933 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:54:11.060967 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:11.060969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:11.060982 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:11.061213 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:54:11.061281 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.062012 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.064194 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.065400 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.065481 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.065538 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:11.065733 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 08:54:11.065774 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":86,"threads_started":1}
I20260928 08:54:11.065872 32177 tablet_service.cc:1467] Processing CreateTablet for tablet 564addc519924983a4f1c3f32637a2f5 (DEFAULT_TABLE table=test-table [id=2a4e947297214cac888c9ea517dbd150]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:54:11.066208 32177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 564addc519924983a4f1c3f32637a2f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:54:11.066619 32037 tablet_service.cc:1467] Processing CreateTablet for tablet 564addc519924983a4f1c3f32637a2f5 (DEFAULT_TABLE table=test-table [id=2a4e947297214cac888c9ea517dbd150]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:54:11.066926 32037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 564addc519924983a4f1c3f32637a2f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:54:11.069731 32268 tablet_bootstrap.cc:492] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b: Bootstrap starting.
I20260928 08:54:11.070137 32270 tablet_bootstrap.cc:492] T 564addc519924983a4f1c3f32637a2f5 P b4a99b7bc9f54355980d72922c77af06: Bootstrap starting.
I20260928 08:54:11.070452 32268 tablet_bootstrap.cc:654] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b: Neither blocks nor log segments found. Creating new log.
I20260928 08:54:11.070719 32270 tablet_bootstrap.cc:654] T 564addc519924983a4f1c3f32637a2f5 P b4a99b7bc9f54355980d72922c77af06: Neither blocks nor log segments found. Creating new log.
I20260928 08:54:11.070761 32268 log.cc:824] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b: Log is configured to *not* fsync() on all Append() calls
I20260928 08:54:11.071020 32270 log.cc:824] T 564addc519924983a4f1c3f32637a2f5 P b4a99b7bc9f54355980d72922c77af06: Log is configured to *not* fsync() on all Append() calls
I20260928 08:54:11.071125 32269 tablet_bootstrap.cc:492] T 564addc519924983a4f1c3f32637a2f5 P 80ec10c0466246e5a4fb75ce61877a1f: Bootstrap starting.
I20260928 08:54:11.071560 32268 tablet_bootstrap.cc:492] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b: No bootstrap required, opened a new log
I20260928 08:54:11.071628 32268 ts_tablet_manager.cc:1397] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:54:11.071774 32270 tablet_bootstrap.cc:492] T 564addc519924983a4f1c3f32637a2f5 P b4a99b7bc9f54355980d72922c77af06: No bootstrap required, opened a new log
I20260928 08:54:11.071784 32269 tablet_bootstrap.cc:654] T 564addc519924983a4f1c3f32637a2f5 P 80ec10c0466246e5a4fb75ce61877a1f: Neither blocks nor log segments found. Creating new log.
I20260928 08:54:11.071841 32270 ts_tablet_manager.cc:1397] T 564addc519924983a4f1c3f32637a2f5 P b4a99b7bc9f54355980d72922c77af06: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:54:11.072031 32269 log.cc:824] T 564addc519924983a4f1c3f32637a2f5 P 80ec10c0466246e5a4fb75ce61877a1f: Log is configured to *not* fsync() on all Append() calls
I20260928 08:54:11.072579 32269 tablet_bootstrap.cc:492] T 564addc519924983a4f1c3f32637a2f5 P 80ec10c0466246e5a4fb75ce61877a1f: No bootstrap required, opened a new log
I20260928 08:54:11.072618 32269 ts_tablet_manager.cc:1397] T 564addc519924983a4f1c3f32637a2f5 P 80ec10c0466246e5a4fb75ce61877a1f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:54:11.073678 32268 raft_consensus.cc:348] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } } peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } }
I20260928 08:54:11.073704 32270 raft_consensus.cc:348] T 564addc519924983a4f1c3f32637a2f5 P b4a99b7bc9f54355980d72922c77af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } } peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } }
I20260928 08:54:11.073849 32268 raft_consensus.cc:374] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:54:11.073849 32270 raft_consensus.cc:374] T 564addc519924983a4f1c3f32637a2f5 P b4a99b7bc9f54355980d72922c77af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:54:11.073884 32268 raft_consensus.cc:729] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7fa0d4b654942aca29b1488c9dacb1b, State: Initialized, Role: FOLLOWER
I20260928 08:54:11.073884 32270 raft_consensus.cc:729] T 564addc519924983a4f1c3f32637a2f5 P b4a99b7bc9f54355980d72922c77af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4a99b7bc9f54355980d72922c77af06, State: Initialized, Role: FOLLOWER
I20260928 08:54:11.073993 32270 consensus_queue.cc:260] T 564addc519924983a4f1c3f32637a2f5 P b4a99b7bc9f54355980d72922c77af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } } peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } }
I20260928 08:54:11.074007 32268 consensus_queue.cc:260] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } } peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } }
I20260928 08:54:11.074127 32269 raft_consensus.cc:348] T 564addc519924983a4f1c3f32637a2f5 P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } } peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } }
I20260928 08:54:11.074237 32268 ts_tablet_manager.cc:1428] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:54:11.074237 32270 ts_tablet_manager.cc:1428] T 564addc519924983a4f1c3f32637a2f5 P b4a99b7bc9f54355980d72922c77af06: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 08:54:11.074258 32269 raft_consensus.cc:374] T 564addc519924983a4f1c3f32637a2f5 P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:54:11.074297 32269 raft_consensus.cc:729] T 564addc519924983a4f1c3f32637a2f5 P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80ec10c0466246e5a4fb75ce61877a1f, State: Initialized, Role: FOLLOWER
I20260928 08:54:11.074350 32103 heartbeater.cc:503] Master 127.23.167.254:42661 was elected leader, sending a full tablet report...
I20260928 08:54:11.074404 32269 consensus_queue.cc:260] T 564addc519924983a4f1c3f32637a2f5 P 80ec10c0466246e5a4fb75ce61877a1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } } peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } }
I20260928 08:54:11.074646 32269 ts_tablet_manager.cc:1428] T 564addc519924983a4f1c3f32637a2f5 P 80ec10c0466246e5a4fb75ce61877a1f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:54:11.074638 31963 heartbeater.cc:503] Master 127.23.167.254:42661 was elected leader, sending a full tablet report...
I20260928 08:54:11.074750 32243 heartbeater.cc:503] Master 127.23.167.254:42661 was elected leader, sending a full tablet report...
W20260928 08:54:11.113431 32104 tablet.cc:2369] T 564addc519924983a4f1c3f32637a2f5 P b4a99b7bc9f54355980d72922c77af06: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:54:11.214615 31964 tablet.cc:2369] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:54:11.257606 32274 raft_consensus.cc:482] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:54:11.257753 32274 raft_consensus.cc:504] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } } peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } }
I20260928 08:54:11.258116 32274 leader_election.cc:288] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4a99b7bc9f54355980d72922c77af06 (127.23.167.194:33563), 80ec10c0466246e5a4fb75ce61877a1f (127.23.167.195:42447)
I20260928 08:54:11.260977 32265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.258399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:52021 (local address 127.23.167.195:42447)
0928 08:54:11.258481 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:11.258483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:11.258622 (+ 139us) server_negotiation.cc:407] Connection header received
0928 08:54:11.258679 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:11.258680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:11.258690 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:11.258745 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:54:11.258881 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.259362 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.260325 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.260680 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.260743 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.260751 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:11.260880 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 08:54:11.260913 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 08:54:11.260978 32267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.258479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:52505 (local address 127.23.167.194:33563)
0928 08:54:11.258543 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:11.258544 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:11.258554 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:54:11.258588 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:11.258589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:11.258599 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:11.258658 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:54:11.258813 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.259339 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.260316 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.260660 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.260715 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.260723 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:11.260866 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 08:54:11.260903 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 08:54:11.260990 32278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.258306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:33563 (local address 127.23.167.193:52505)
0928 08:54:11.258462 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:11.258474 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:11.258532 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:11.258676 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:11.258676 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:11.258679 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:54:11.258749 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.258751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.259420 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.259421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.260230 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.260232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.260735 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.260736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.260799 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.260804 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 08:54:11.260851 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 08:54:11.260891 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":3,"thread_start_us":71,"threads_started":1}
I20260928 08:54:11.260995 32277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.258306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:42447 (local address 127.23.167.193:52021)
0928 08:54:11.258540 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:11.258551 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:11.258616 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:11.258752 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:11.258752 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:11.258754 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:54:11.258817 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.258818 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.259446 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.259447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.260231 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.260233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.260753 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.260753 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.260813 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.260816 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:54:11.260864 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 08:54:11.260904 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":55,"thread_start_us":73,"threads_started":1}
I20260928 08:54:11.261343 32057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564addc519924983a4f1c3f32637a2f5" candidate_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a99b7bc9f54355980d72922c77af06" is_pre_election: true
I20260928 08:54:11.261503 32057 raft_consensus.cc:2454] T 564addc519924983a4f1c3f32637a2f5 P b4a99b7bc9f54355980d72922c77af06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7fa0d4b654942aca29b1488c9dacb1b in term 0.
I20260928 08:54:11.261559 32197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564addc519924983a4f1c3f32637a2f5" candidate_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80ec10c0466246e5a4fb75ce61877a1f" is_pre_election: true
I20260928 08:54:11.261703 32197 raft_consensus.cc:2454] T 564addc519924983a4f1c3f32637a2f5 P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7fa0d4b654942aca29b1488c9dacb1b in term 0.
I20260928 08:54:11.261910 31849 leader_election.cc:302] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4a99b7bc9f54355980d72922c77af06, c7fa0d4b654942aca29b1488c9dacb1b; no voters:
I20260928 08:54:11.262027 32274 raft_consensus.cc:2781] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:54:11.262074 32274 raft_consensus.cc:482] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:54:11.262102 32274 raft_consensus.cc:3037] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:54:11.262751 32274 raft_consensus.cc:504] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } } peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } }
I20260928 08:54:11.262871 32274 leader_election.cc:288] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b [CANDIDATE]: Term 1 election: Requested vote from peers b4a99b7bc9f54355980d72922c77af06 (127.23.167.194:33563), 80ec10c0466246e5a4fb75ce61877a1f (127.23.167.195:42447)
I20260928 08:54:11.263006 32197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564addc519924983a4f1c3f32637a2f5" candidate_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80ec10c0466246e5a4fb75ce61877a1f"
I20260928 08:54:11.263016 32057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564addc519924983a4f1c3f32637a2f5" candidate_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a99b7bc9f54355980d72922c77af06"
I20260928 08:54:11.263080 32057 raft_consensus.cc:3037] T 564addc519924983a4f1c3f32637a2f5 P b4a99b7bc9f54355980d72922c77af06 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:54:11.263083 32197 raft_consensus.cc:3037] T 564addc519924983a4f1c3f32637a2f5 P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:54:11.263721 32057 raft_consensus.cc:2454] T 564addc519924983a4f1c3f32637a2f5 P b4a99b7bc9f54355980d72922c77af06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7fa0d4b654942aca29b1488c9dacb1b in term 1.
I20260928 08:54:11.263741 32197 raft_consensus.cc:2454] T 564addc519924983a4f1c3f32637a2f5 P 80ec10c0466246e5a4fb75ce61877a1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7fa0d4b654942aca29b1488c9dacb1b in term 1.
I20260928 08:54:11.263880 31848 leader_election.cc:302] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80ec10c0466246e5a4fb75ce61877a1f, c7fa0d4b654942aca29b1488c9dacb1b; no voters:
I20260928 08:54:11.264011 32274 raft_consensus.cc:2781] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:54:11.264132 32274 raft_consensus.cc:686] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b [term 1 LEADER]: Becoming Leader. State: Replica: c7fa0d4b654942aca29b1488c9dacb1b, State: Running, Role: LEADER
I20260928 08:54:11.264221 32274 consensus_queue.cc:237] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } } peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } }
I20260928 08:54:11.264999 31758 catalog_manager.cc:5495] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b reported cstate change: term changed from 0 to 1, leader changed from <none> to c7fa0d4b654942aca29b1488c9dacb1b (127.23.167.193). New cstate: current_term: 1 leader_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } health_report { overall_health: HEALTHY } } }
W20260928 08:54:11.267095 32244 tablet.cc:2369] T 564addc519924983a4f1c3f32637a2f5 P 80ec10c0466246e5a4fb75ce61877a1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:54:11.289655 32266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.287309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37298 (local address 127.23.167.193:46433)
0928 08:54:11.287369 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:11.287371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:11.287379 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:54:11.287443 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:11.287444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:11.287467 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:11.287535 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:54:11.287654 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.288140 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.288917 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.289146 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.289200 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.289350 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:54:11.289460 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:54:11.289536 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:11.289554 (+ 18us) server_negotiation.cc:299] Negotiation successful
0928 08:54:11.289576 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 08:54:11.291510 32197 raft_consensus.cc:1261] T 564addc519924983a4f1c3f32637a2f5 P 80ec10c0466246e5a4fb75ce61877a1f [term 1 FOLLOWER]: Refusing update from remote peer c7fa0d4b654942aca29b1488c9dacb1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:54:11.291558 32057 raft_consensus.cc:1261] T 564addc519924983a4f1c3f32637a2f5 P b4a99b7bc9f54355980d72922c77af06 [term 1 FOLLOWER]: Refusing update from remote peer c7fa0d4b654942aca29b1488c9dacb1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:54:11.291836 32279 consensus_queue.cc:1035] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:54:11.291973 32274 consensus_queue.cc:1035] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:54:11.294147 32282 mvcc.cc:205] Tried to move back new op lower bound from 7334238827687211008 to 7334238827578933248. Current Snapshot: MvccSnapshot[applied={T|T < 7334238827687211008}]
I20260928 08:54:11.294523 32283 mvcc.cc:205] Tried to move back new op lower bound from 7334238827687211008 to 7334238827578933248. Current Snapshot: MvccSnapshot[applied={T|T < 7334238827687211008}]
I20260928 08:54:11.296682 31760 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:54:11.296762 31760 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:54:11.297453 31760 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc1b95ca7c2d48d5ae1c2c1064bcb2b1: Sending DeleteTablet for 3 replicas of tablet 564addc519924983a4f1c3f32637a2f5
I20260928 08:54:11.297739 32037 tablet_service.cc:1514] Processing DeleteTablet for tablet 564addc519924983a4f1c3f32637a2f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:54:11 UTC) from {username='oryx', principal='oryx/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:56940
I20260928 08:54:11.297870 32177 tablet_service.cc:1514] Processing DeleteTablet for tablet 564addc519924983a4f1c3f32637a2f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:54:11 UTC) from {username='oryx', principal='oryx/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:46584
I20260928 08:54:11.298086 32293 tablet_replica.cc:330] T 564addc519924983a4f1c3f32637a2f5 P 80ec10c0466246e5a4fb75ce61877a1f: stopping tablet replica
I20260928 08:54:11.298009 31760 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 08:54:11.298089 31897 tablet_service.cc:1514] Processing DeleteTablet for tablet 564addc519924983a4f1c3f32637a2f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:54:11 UTC) from {username='oryx', principal='oryx/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:37296
W20260928 08:54:11.298162 31760 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:54:11.298265 32293 raft_consensus.cc:2229] T 564addc519924983a4f1c3f32637a2f5 P 80ec10c0466246e5a4fb75ce61877a1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:54:11.298415 32294 tablet_replica.cc:330] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b: stopping tablet replica
I20260928 08:54:11.298543 32294 raft_consensus.cc:2229] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:54:11.298691 32293 raft_consensus.cc:2258] T 564addc519924983a4f1c3f32637a2f5 P 80ec10c0466246e5a4fb75ce61877a1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:54:11.299590 32037 tablet_service.cc:1467] Processing CreateTablet for tablet 88c9d4be3bb241a09cfb087921b75e77 (DEFAULT_TABLE table=test-table [id=dcf39fccf00545279173f9df153a6930]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:54:11.299705 32037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88c9d4be3bb241a09cfb087921b75e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:54:11.300750 32270 tablet_bootstrap.cc:492] T 88c9d4be3bb241a09cfb087921b75e77 P b4a99b7bc9f54355980d72922c77af06: Bootstrap starting.
I20260928 08:54:11.300930 32177 tablet_service.cc:1467] Processing CreateTablet for tablet 88c9d4be3bb241a09cfb087921b75e77 (DEFAULT_TABLE table=test-table [id=dcf39fccf00545279173f9df153a6930]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:54:11.301028 32177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88c9d4be3bb241a09cfb087921b75e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:54:11.301189 32270 tablet_bootstrap.cc:654] T 88c9d4be3bb241a09cfb087921b75e77 P b4a99b7bc9f54355980d72922c77af06: Neither blocks nor log segments found. Creating new log.
I20260928 08:54:11.302088 32269 tablet_bootstrap.cc:492] T 88c9d4be3bb241a09cfb087921b75e77 P 80ec10c0466246e5a4fb75ce61877a1f: Bootstrap starting.
I20260928 08:54:11.302489 32269 tablet_bootstrap.cc:654] T 88c9d4be3bb241a09cfb087921b75e77 P 80ec10c0466246e5a4fb75ce61877a1f: Neither blocks nor log segments found. Creating new log.
I20260928 08:54:11.302656 32293 ts_tablet_manager.cc:1905] T 564addc519924983a4f1c3f32637a2f5 P 80ec10c0466246e5a4fb75ce61877a1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:54:11.302971 31897 tablet_service.cc:1467] Processing CreateTablet for tablet 88c9d4be3bb241a09cfb087921b75e77 (DEFAULT_TABLE table=test-table [id=dcf39fccf00545279173f9df153a6930]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:54:11.303339 31897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88c9d4be3bb241a09cfb087921b75e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:54:11.304313 32293 ts_tablet_manager.cc:1918] T 564addc519924983a4f1c3f32637a2f5 P 80ec10c0466246e5a4fb75ce61877a1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:54:11.304383 32293 log.cc:1196] T 564addc519924983a4f1c3f32637a2f5 P 80ec10c0466246e5a4fb75ce61877a1f: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-2/wal/wals/564addc519924983a4f1c3f32637a2f5
I20260928 08:54:11.304639 32293 ts_tablet_manager.cc:1939] T 564addc519924983a4f1c3f32637a2f5 P 80ec10c0466246e5a4fb75ce61877a1f: Deleting consensus metadata
I20260928 08:54:11.305014 31744 catalog_manager.cc:4841] TS 80ec10c0466246e5a4fb75ce61877a1f (127.23.167.195:42447): tablet 564addc519924983a4f1c3f32637a2f5 (table test-table [id=2a4e947297214cac888c9ea517dbd150]) successfully deleted
I20260928 08:54:11.305449 32270 tablet_bootstrap.cc:492] T 88c9d4be3bb241a09cfb087921b75e77 P b4a99b7bc9f54355980d72922c77af06: No bootstrap required, opened a new log
I20260928 08:54:11.305455 32268 tablet_bootstrap.cc:492] T 88c9d4be3bb241a09cfb087921b75e77 P c7fa0d4b654942aca29b1488c9dacb1b: Bootstrap starting.
I20260928 08:54:11.305492 32270 ts_tablet_manager.cc:1397] T 88c9d4be3bb241a09cfb087921b75e77 P b4a99b7bc9f54355980d72922c77af06: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 08:54:11.305615 32294 raft_consensus.cc:2258] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:54:11.305657 32270 raft_consensus.cc:348] T 88c9d4be3bb241a09cfb087921b75e77 P b4a99b7bc9f54355980d72922c77af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } }
I20260928 08:54:11.305732 32270 raft_consensus.cc:374] T 88c9d4be3bb241a09cfb087921b75e77 P b4a99b7bc9f54355980d72922c77af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:54:11.305749 32270 raft_consensus.cc:729] T 88c9d4be3bb241a09cfb087921b75e77 P b4a99b7bc9f54355980d72922c77af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4a99b7bc9f54355980d72922c77af06, State: Initialized, Role: FOLLOWER
I20260928 08:54:11.305792 32270 consensus_queue.cc:260] T 88c9d4be3bb241a09cfb087921b75e77 P b4a99b7bc9f54355980d72922c77af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } }
I20260928 08:54:11.305857 32268 tablet_bootstrap.cc:654] T 88c9d4be3bb241a09cfb087921b75e77 P c7fa0d4b654942aca29b1488c9dacb1b: Neither blocks nor log segments found. Creating new log.
I20260928 08:54:11.305881 32270 ts_tablet_manager.cc:1428] T 88c9d4be3bb241a09cfb087921b75e77 P b4a99b7bc9f54355980d72922c77af06: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:11.306636 32292 tablet_replica.cc:330] T 564addc519924983a4f1c3f32637a2f5 P b4a99b7bc9f54355980d72922c77af06: stopping tablet replica
I20260928 08:54:11.306753 32292 raft_consensus.cc:2229] T 564addc519924983a4f1c3f32637a2f5 P b4a99b7bc9f54355980d72922c77af06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:54:11.306867 32292 raft_consensus.cc:2258] T 564addc519924983a4f1c3f32637a2f5 P b4a99b7bc9f54355980d72922c77af06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:54:11.307762 32294 ts_tablet_manager.cc:1905] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:54:11.309433 32292 ts_tablet_manager.cc:1905] T 564addc519924983a4f1c3f32637a2f5 P b4a99b7bc9f54355980d72922c77af06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:54:11.309607 32294 ts_tablet_manager.cc:1918] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:54:11.309676 32294 log.cc:1196] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-0/wal/wals/564addc519924983a4f1c3f32637a2f5
I20260928 08:54:11.309903 32294 ts_tablet_manager.cc:1939] T 564addc519924983a4f1c3f32637a2f5 P c7fa0d4b654942aca29b1488c9dacb1b: Deleting consensus metadata
I20260928 08:54:11.310308 31746 catalog_manager.cc:4841] TS c7fa0d4b654942aca29b1488c9dacb1b (127.23.167.193:46433): tablet 564addc519924983a4f1c3f32637a2f5 (table test-table [id=2a4e947297214cac888c9ea517dbd150]) successfully deleted
I20260928 08:54:11.311147 32292 ts_tablet_manager.cc:1918] T 564addc519924983a4f1c3f32637a2f5 P b4a99b7bc9f54355980d72922c77af06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:54:11.311204 32292 log.cc:1196] T 564addc519924983a4f1c3f32637a2f5 P b4a99b7bc9f54355980d72922c77af06: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-1/wal/wals/564addc519924983a4f1c3f32637a2f5
I20260928 08:54:11.311484 32292 ts_tablet_manager.cc:1939] T 564addc519924983a4f1c3f32637a2f5 P b4a99b7bc9f54355980d72922c77af06: Deleting consensus metadata
I20260928 08:54:11.311837 31745 catalog_manager.cc:4841] TS b4a99b7bc9f54355980d72922c77af06 (127.23.167.194:33563): tablet 564addc519924983a4f1c3f32637a2f5 (table test-table [id=2a4e947297214cac888c9ea517dbd150]) successfully deleted
I20260928 08:54:11.313407 32268 tablet_bootstrap.cc:492] T 88c9d4be3bb241a09cfb087921b75e77 P c7fa0d4b654942aca29b1488c9dacb1b: No bootstrap required, opened a new log
I20260928 08:54:11.313452 32268 ts_tablet_manager.cc:1397] T 88c9d4be3bb241a09cfb087921b75e77 P c7fa0d4b654942aca29b1488c9dacb1b: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260928 08:54:11.313568 32268 raft_consensus.cc:348] T 88c9d4be3bb241a09cfb087921b75e77 P c7fa0d4b654942aca29b1488c9dacb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } }
I20260928 08:54:11.313625 32268 raft_consensus.cc:374] T 88c9d4be3bb241a09cfb087921b75e77 P c7fa0d4b654942aca29b1488c9dacb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:54:11.313637 32268 raft_consensus.cc:729] T 88c9d4be3bb241a09cfb087921b75e77 P c7fa0d4b654942aca29b1488c9dacb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7fa0d4b654942aca29b1488c9dacb1b, State: Initialized, Role: FOLLOWER
I20260928 08:54:11.313670 32268 consensus_queue.cc:260] T 88c9d4be3bb241a09cfb087921b75e77 P c7fa0d4b654942aca29b1488c9dacb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } }
I20260928 08:54:11.313750 32268 ts_tablet_manager.cc:1428] T 88c9d4be3bb241a09cfb087921b75e77 P c7fa0d4b654942aca29b1488c9dacb1b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:11.313951 32269 tablet_bootstrap.cc:492] T 88c9d4be3bb241a09cfb087921b75e77 P 80ec10c0466246e5a4fb75ce61877a1f: No bootstrap required, opened a new log
I20260928 08:54:11.314004 32269 ts_tablet_manager.cc:1397] T 88c9d4be3bb241a09cfb087921b75e77 P 80ec10c0466246e5a4fb75ce61877a1f: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20260928 08:54:11.314158 32269 raft_consensus.cc:348] T 88c9d4be3bb241a09cfb087921b75e77 P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } }
I20260928 08:54:11.314229 32269 raft_consensus.cc:374] T 88c9d4be3bb241a09cfb087921b75e77 P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:54:11.314254 32269 raft_consensus.cc:729] T 88c9d4be3bb241a09cfb087921b75e77 P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80ec10c0466246e5a4fb75ce61877a1f, State: Initialized, Role: FOLLOWER
I20260928 08:54:11.314291 32269 consensus_queue.cc:260] T 88c9d4be3bb241a09cfb087921b75e77 P 80ec10c0466246e5a4fb75ce61877a1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } }
I20260928 08:54:11.314378 32269 ts_tablet_manager.cc:1428] T 88c9d4be3bb241a09cfb087921b75e77 P 80ec10c0466246e5a4fb75ce61877a1f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 08:54:11.395968 32287 raft_consensus.cc:482] T 88c9d4be3bb241a09cfb087921b75e77 P c7fa0d4b654942aca29b1488c9dacb1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:54:11.396075 32287 raft_consensus.cc:504] T 88c9d4be3bb241a09cfb087921b75e77 P c7fa0d4b654942aca29b1488c9dacb1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } }
I20260928 08:54:11.396250 32287 leader_election.cc:288] T 88c9d4be3bb241a09cfb087921b75e77 P c7fa0d4b654942aca29b1488c9dacb1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4a99b7bc9f54355980d72922c77af06 (127.23.167.194:33563), 80ec10c0466246e5a4fb75ce61877a1f (127.23.167.195:42447)
I20260928 08:54:11.396468 32057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88c9d4be3bb241a09cfb087921b75e77" candidate_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a99b7bc9f54355980d72922c77af06" is_pre_election: true
I20260928 08:54:11.396462 32197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88c9d4be3bb241a09cfb087921b75e77" candidate_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80ec10c0466246e5a4fb75ce61877a1f" is_pre_election: true
I20260928 08:54:11.396569 32057 raft_consensus.cc:2454] T 88c9d4be3bb241a09cfb087921b75e77 P b4a99b7bc9f54355980d72922c77af06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7fa0d4b654942aca29b1488c9dacb1b in term 0.
I20260928 08:54:11.396569 32197 raft_consensus.cc:2454] T 88c9d4be3bb241a09cfb087921b75e77 P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7fa0d4b654942aca29b1488c9dacb1b in term 0.
I20260928 08:54:11.396718 31849 leader_election.cc:302] T 88c9d4be3bb241a09cfb087921b75e77 P c7fa0d4b654942aca29b1488c9dacb1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4a99b7bc9f54355980d72922c77af06, c7fa0d4b654942aca29b1488c9dacb1b; no voters:
I20260928 08:54:11.396818 32287 raft_consensus.cc:2781] T 88c9d4be3bb241a09cfb087921b75e77 P c7fa0d4b654942aca29b1488c9dacb1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:54:11.396852 32287 raft_consensus.cc:482] T 88c9d4be3bb241a09cfb087921b75e77 P c7fa0d4b654942aca29b1488c9dacb1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:54:11.396872 32287 raft_consensus.cc:3037] T 88c9d4be3bb241a09cfb087921b75e77 P c7fa0d4b654942aca29b1488c9dacb1b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:54:11.397712 32287 raft_consensus.cc:504] T 88c9d4be3bb241a09cfb087921b75e77 P c7fa0d4b654942aca29b1488c9dacb1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } }
I20260928 08:54:11.397835 32287 leader_election.cc:288] T 88c9d4be3bb241a09cfb087921b75e77 P c7fa0d4b654942aca29b1488c9dacb1b [CANDIDATE]: Term 1 election: Requested vote from peers b4a99b7bc9f54355980d72922c77af06 (127.23.167.194:33563), 80ec10c0466246e5a4fb75ce61877a1f (127.23.167.195:42447)
I20260928 08:54:11.398011 32197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88c9d4be3bb241a09cfb087921b75e77" candidate_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80ec10c0466246e5a4fb75ce61877a1f"
I20260928 08:54:11.398041 32057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88c9d4be3bb241a09cfb087921b75e77" candidate_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a99b7bc9f54355980d72922c77af06"
I20260928 08:54:11.398090 32197 raft_consensus.cc:3037] T 88c9d4be3bb241a09cfb087921b75e77 P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:54:11.398113 32057 raft_consensus.cc:3037] T 88c9d4be3bb241a09cfb087921b75e77 P b4a99b7bc9f54355980d72922c77af06 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:54:11.398691 32057 raft_consensus.cc:2454] T 88c9d4be3bb241a09cfb087921b75e77 P b4a99b7bc9f54355980d72922c77af06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7fa0d4b654942aca29b1488c9dacb1b in term 1.
I20260928 08:54:11.398851 31849 leader_election.cc:302] T 88c9d4be3bb241a09cfb087921b75e77 P c7fa0d4b654942aca29b1488c9dacb1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4a99b7bc9f54355980d72922c77af06, c7fa0d4b654942aca29b1488c9dacb1b; no voters:
I20260928 08:54:11.398866 32197 raft_consensus.cc:2454] T 88c9d4be3bb241a09cfb087921b75e77 P 80ec10c0466246e5a4fb75ce61877a1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7fa0d4b654942aca29b1488c9dacb1b in term 1.
I20260928 08:54:11.398927 32287 raft_consensus.cc:2781] T 88c9d4be3bb241a09cfb087921b75e77 P c7fa0d4b654942aca29b1488c9dacb1b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:54:11.399016 32287 raft_consensus.cc:686] T 88c9d4be3bb241a09cfb087921b75e77 P c7fa0d4b654942aca29b1488c9dacb1b [term 1 LEADER]: Becoming Leader. State: Replica: c7fa0d4b654942aca29b1488c9dacb1b, State: Running, Role: LEADER
I20260928 08:54:11.399063 32287 consensus_queue.cc:237] T 88c9d4be3bb241a09cfb087921b75e77 P c7fa0d4b654942aca29b1488c9dacb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } }
I20260928 08:54:11.399633 31759 catalog_manager.cc:5495] T 88c9d4be3bb241a09cfb087921b75e77 P c7fa0d4b654942aca29b1488c9dacb1b reported cstate change: term changed from 0 to 1, leader changed from <none> to c7fa0d4b654942aca29b1488c9dacb1b (127.23.167.193). New cstate: current_term: 1 leader_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } health_report { overall_health: UNKNOWN } } }
Sep 28 08:54:11 dist-test-slave-pbjp krb5kdc[31715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585650, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.254@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM
I20260928 08:54:11.452260 32304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.445784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42661 (local address 127.0.0.1:35412)
0928 08:54:11.445930 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:11.445941 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:11.446014 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:11.446131 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:11.446132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:11.446488 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:11.446541 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.446545 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.447193 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.447194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.447726 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.447728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.448079 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.448080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.448133 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.448595 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:11.448600 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:11.449968 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:54:11.451380 (+ 1412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:54:11.451386 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:54:11.451388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:54:11.451608 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:54:11.451749 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:54:11.451749 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:54:11.451750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:54:11.451798 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:54:11.451977 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:54:11.451981 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:54:11.452019 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 08:54:11.452064 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 08:54:11.452162 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":3,"thread_start_us":76,"threads_started":1}
I20260928 08:54:11.452302 31810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.445848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35412 (local address 127.23.167.254:42661)
0928 08:54:11.445911 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:11.445913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:11.446008 (+ 95us) server_negotiation.cc:407] Connection header received
0928 08:54:11.446063 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:11.446064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:11.446074 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:11.446119 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:11.446633 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.447120 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.447808 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.448016 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.448072 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.450055 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:54:11.450057 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:54:11.450057 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:54:11.450064 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:54:11.451312 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:11.451661 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:11.451662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:11.451662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:11.451705 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:11.451840 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:11.451840 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:11.451841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:11.451937 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:54:11.451981 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:11.452081 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 08:54:11.452202 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":37}
I20260928 08:54:11.454116 31759 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:35412:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 08:54:11.454308 31759 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:54:11.455606 31897 tablet_service.cc:1467] Processing CreateTablet for tablet a36e512ec53b46249b5bbcac586fc442 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6eefdd9e347745f0b776783545b9d359]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:54:11.455598 32177 tablet_service.cc:1467] Processing CreateTablet for tablet a36e512ec53b46249b5bbcac586fc442 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6eefdd9e347745f0b776783545b9d359]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:54:11.455657 32037 tablet_service.cc:1467] Processing CreateTablet for tablet a36e512ec53b46249b5bbcac586fc442 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6eefdd9e347745f0b776783545b9d359]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:54:11.455754 32177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a36e512ec53b46249b5bbcac586fc442. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:54:11.455755 31897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a36e512ec53b46249b5bbcac586fc442. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:54:11.455798 32037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a36e512ec53b46249b5bbcac586fc442. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:54:11.456869 32270 tablet_bootstrap.cc:492] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06: Bootstrap starting.
I20260928 08:54:11.457202 32268 tablet_bootstrap.cc:492] T a36e512ec53b46249b5bbcac586fc442 P c7fa0d4b654942aca29b1488c9dacb1b: Bootstrap starting.
I20260928 08:54:11.457286 32270 tablet_bootstrap.cc:654] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06: Neither blocks nor log segments found. Creating new log.
I20260928 08:54:11.457507 32268 tablet_bootstrap.cc:654] T a36e512ec53b46249b5bbcac586fc442 P c7fa0d4b654942aca29b1488c9dacb1b: Neither blocks nor log segments found. Creating new log.
I20260928 08:54:11.457732 32269 tablet_bootstrap.cc:492] T a36e512ec53b46249b5bbcac586fc442 P 80ec10c0466246e5a4fb75ce61877a1f: Bootstrap starting.
I20260928 08:54:11.457952 32270 tablet_bootstrap.cc:492] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06: No bootstrap required, opened a new log
I20260928 08:54:11.458001 32270 ts_tablet_manager.cc:1397] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 08:54:11.458009 32268 tablet_bootstrap.cc:492] T a36e512ec53b46249b5bbcac586fc442 P c7fa0d4b654942aca29b1488c9dacb1b: No bootstrap required, opened a new log
I20260928 08:54:11.458050 32268 ts_tablet_manager.cc:1397] T a36e512ec53b46249b5bbcac586fc442 P c7fa0d4b654942aca29b1488c9dacb1b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 08:54:11.458104 32269 tablet_bootstrap.cc:654] T a36e512ec53b46249b5bbcac586fc442 P 80ec10c0466246e5a4fb75ce61877a1f: Neither blocks nor log segments found. Creating new log.
I20260928 08:54:11.458150 32270 raft_consensus.cc:348] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } }
I20260928 08:54:11.458165 32268 raft_consensus.cc:348] T a36e512ec53b46249b5bbcac586fc442 P c7fa0d4b654942aca29b1488c9dacb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } }
I20260928 08:54:11.458232 32268 raft_consensus.cc:374] T a36e512ec53b46249b5bbcac586fc442 P c7fa0d4b654942aca29b1488c9dacb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:54:11.458233 32270 raft_consensus.cc:374] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:54:11.458252 32268 raft_consensus.cc:729] T a36e512ec53b46249b5bbcac586fc442 P c7fa0d4b654942aca29b1488c9dacb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7fa0d4b654942aca29b1488c9dacb1b, State: Initialized, Role: FOLLOWER
I20260928 08:54:11.458254 32270 raft_consensus.cc:729] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4a99b7bc9f54355980d72922c77af06, State: Initialized, Role: FOLLOWER
I20260928 08:54:11.458294 32268 consensus_queue.cc:260] T a36e512ec53b46249b5bbcac586fc442 P c7fa0d4b654942aca29b1488c9dacb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } }
I20260928 08:54:11.458297 32270 consensus_queue.cc:260] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } }
I20260928 08:54:11.458377 32268 ts_tablet_manager.cc:1428] T a36e512ec53b46249b5bbcac586fc442 P c7fa0d4b654942aca29b1488c9dacb1b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:11.458379 32270 ts_tablet_manager.cc:1428] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 08:54:11.458493 32287 tablet_replica.cc:439] T a36e512ec53b46249b5bbcac586fc442 P c7fa0d4b654942aca29b1488c9dacb1b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } }
I20260928 08:54:11.458501 32275 tablet_replica.cc:439] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } }
I20260928 08:54:11.458554 32287 tablet_replica.cc:442] T a36e512ec53b46249b5bbcac586fc442 P c7fa0d4b654942aca29b1488c9dacb1b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:54:11.458562 32275 tablet_replica.cc:442] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:54:11.459112 32269 tablet_bootstrap.cc:492] T a36e512ec53b46249b5bbcac586fc442 P 80ec10c0466246e5a4fb75ce61877a1f: No bootstrap required, opened a new log
I20260928 08:54:11.459157 32269 ts_tablet_manager.cc:1397] T a36e512ec53b46249b5bbcac586fc442 P 80ec10c0466246e5a4fb75ce61877a1f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 08:54:11.459321 32269 raft_consensus.cc:348] T a36e512ec53b46249b5bbcac586fc442 P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } }
I20260928 08:54:11.459383 32269 raft_consensus.cc:374] T a36e512ec53b46249b5bbcac586fc442 P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:54:11.459395 32269 raft_consensus.cc:729] T a36e512ec53b46249b5bbcac586fc442 P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80ec10c0466246e5a4fb75ce61877a1f, State: Initialized, Role: FOLLOWER
I20260928 08:54:11.459437 32269 consensus_queue.cc:260] T a36e512ec53b46249b5bbcac586fc442 P 80ec10c0466246e5a4fb75ce61877a1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } }
I20260928 08:54:11.459522 32269 ts_tablet_manager.cc:1428] T a36e512ec53b46249b5bbcac586fc442 P 80ec10c0466246e5a4fb75ce61877a1f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:11.459604 32276 tablet_replica.cc:439] T a36e512ec53b46249b5bbcac586fc442 P 80ec10c0466246e5a4fb75ce61877a1f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } }
I20260928 08:54:11.459697 32276 tablet_replica.cc:442] T a36e512ec53b46249b5bbcac586fc442 P 80ec10c0466246e5a4fb75ce61877a1f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:54:11.626099 32276 raft_consensus.cc:482] T a36e512ec53b46249b5bbcac586fc442 P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:54:11.626188 32276 raft_consensus.cc:504] T a36e512ec53b46249b5bbcac586fc442 P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } }
I20260928 08:54:11.626482 32276 leader_election.cc:288] T a36e512ec53b46249b5bbcac586fc442 P 80ec10c0466246e5a4fb75ce61877a1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7fa0d4b654942aca29b1488c9dacb1b (127.23.167.193:46433), b4a99b7bc9f54355980d72922c77af06 (127.23.167.194:33563)
I20260928 08:54:11.627096 32275 raft_consensus.cc:482] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:54:11.627166 32275 raft_consensus.cc:504] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } }
I20260928 08:54:11.627501 32275 leader_election.cc:288] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7fa0d4b654942aca29b1488c9dacb1b (127.23.167.193:46433), 80ec10c0466246e5a4fb75ce61877a1f (127.23.167.195:42447)
I20260928 08:54:11.630592 32266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.626737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:60469 (local address 127.23.167.193:46433)
0928 08:54:11.626787 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:11.626790 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:11.626888 (+ 98us) server_negotiation.cc:407] Connection header received
0928 08:54:11.626967 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:11.626968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:11.626979 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:11.627030 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:54:11.627213 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.627960 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.629544 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.630181 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.630223 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.630233 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:11.630483 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 08:54:11.630523 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 08:54:11.630862 32267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.626766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:56367 (local address 127.23.167.194:33563)
0928 08:54:11.626816 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:11.626818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:11.626888 (+ 70us) server_negotiation.cc:407] Connection header received
0928 08:54:11.626953 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:11.626954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:11.626963 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:11.627025 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:54:11.627212 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.627973 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.630014 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.630513 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.630553 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.630560 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:11.630766 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 08:54:11.630804 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 08:54:11.630888 32308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.626669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:46433 (local address 127.23.167.195:60469)
0928 08:54:11.626827 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:11.626836 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:11.626893 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:11.627084 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:11.627085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:11.627086 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:54:11.627130 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.627132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.628059 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.628060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.629450 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.629453 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.630286 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.630287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.630407 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.630411 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:54:11.630701 (+ 290us) client_negotiation.cc:240] Negotiation successful
0928 08:54:11.630744 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":28,"thread_start_us":63,"threads_started":1}
I20260928 08:54:11.631072 32309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.626662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:33563 (local address 127.23.167.195:56367)
0928 08:54:11.626799 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:11.626821 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:11.626893 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:11.627042 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:11.627043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:11.627045 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:54:11.627123 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.627125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.628059 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.628060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.629918 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.629921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.630611 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.630612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.630688 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.630692 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:54:11.630942 (+ 250us) client_negotiation.cc:240] Negotiation successful
0928 08:54:11.630986 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":686,"thread_start_us":70,"threads_started":1}
I20260928 08:54:11.631160 32311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.627609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:46433 (local address 127.23.167.194:39987)
0928 08:54:11.627881 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:11.627890 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:11.627947 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:11.628108 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:11.628109 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:11.628110 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:54:11.628179 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.628182 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.629036 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.629037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.630168 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.630171 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.630917 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.630918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.630995 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.630999 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:54:11.631055 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 08:54:11.631091 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":47,"threads_started":1}
I20260928 08:54:11.631170 31917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a36e512ec53b46249b5bbcac586fc442" candidate_uuid: "80ec10c0466246e5a4fb75ce61877a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" is_pre_election: true
I20260928 08:54:11.631294 31917 raft_consensus.cc:2454] T a36e512ec53b46249b5bbcac586fc442 P c7fa0d4b654942aca29b1488c9dacb1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80ec10c0466246e5a4fb75ce61877a1f in term 0.
I20260928 08:54:11.631417 32312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.627753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:39987 (local address 127.23.167.193:46433)
0928 08:54:11.627841 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:11.627843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:11.627933 (+ 90us) server_negotiation.cc:407] Connection header received
0928 08:54:11.628009 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:11.628010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:11.628020 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:11.628071 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:54:11.628237 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.628945 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.630259 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.630741 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.630802 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.630808 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:11.631241 (+ 433us) server_negotiation.cc:299] Negotiation successful
0928 08:54:11.631289 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72,"thread_start_us":39,"threads_started":1}
I20260928 08:54:11.631505 32057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a36e512ec53b46249b5bbcac586fc442" candidate_uuid: "80ec10c0466246e5a4fb75ce61877a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a99b7bc9f54355980d72922c77af06" is_pre_election: true
I20260928 08:54:11.631575 32057 raft_consensus.cc:2454] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80ec10c0466246e5a4fb75ce61877a1f in term 0.
I20260928 08:54:11.631599 31917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a36e512ec53b46249b5bbcac586fc442" candidate_uuid: "b4a99b7bc9f54355980d72922c77af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" is_pre_election: true
I20260928 08:54:11.631664 31917 raft_consensus.cc:2454] T a36e512ec53b46249b5bbcac586fc442 P c7fa0d4b654942aca29b1488c9dacb1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4a99b7bc9f54355980d72922c77af06 in term 0.
I20260928 08:54:11.631698 32265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.627760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:59423 (local address 127.23.167.195:42447)
0928 08:54:11.627796 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:11.627798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:11.628040 (+ 242us) server_negotiation.cc:407] Connection header received
0928 08:54:11.628125 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:11.628125 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:11.628133 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:11.628160 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:54:11.628329 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.628781 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.630182 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.630675 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.630710 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.630717 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:11.631604 (+ 887us) server_negotiation.cc:299] Negotiation successful
0928 08:54:11.631645 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 08:54:11.631785 32130 leader_election.cc:302] T a36e512ec53b46249b5bbcac586fc442 P 80ec10c0466246e5a4fb75ce61877a1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80ec10c0466246e5a4fb75ce61877a1f, c7fa0d4b654942aca29b1488c9dacb1b; no voters:
I20260928 08:54:11.631920 32276 raft_consensus.cc:2781] T a36e512ec53b46249b5bbcac586fc442 P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:54:11.631884 31990 leader_election.cc:302] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4a99b7bc9f54355980d72922c77af06, c7fa0d4b654942aca29b1488c9dacb1b; no voters:
I20260928 08:54:11.631961 32276 raft_consensus.cc:482] T a36e512ec53b46249b5bbcac586fc442 P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:54:11.631987 32276 raft_consensus.cc:3037] T a36e512ec53b46249b5bbcac586fc442 P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:54:11.632009 32275 raft_consensus.cc:2781] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:54:11.632046 32275 raft_consensus.cc:482] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:54:11.632066 32275 raft_consensus.cc:3037] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:54:11.632216 32310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.627644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:42447 (local address 127.23.167.194:59423)
0928 08:54:11.627986 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:11.627994 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:11.628060 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:11.628212 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:11.628212 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:11.628214 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:54:11.628260 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.628261 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.628852 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.628853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.630100 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.630102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.631349 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.631350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.631427 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.631431 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:54:11.632108 (+ 677us) client_negotiation.cc:240] Negotiation successful
0928 08:54:11.632148 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":50,"threads_started":1}
I20260928 08:54:11.632457 32197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a36e512ec53b46249b5bbcac586fc442" candidate_uuid: "b4a99b7bc9f54355980d72922c77af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80ec10c0466246e5a4fb75ce61877a1f" is_pre_election: true
I20260928 08:54:11.632822 32276 raft_consensus.cc:504] T a36e512ec53b46249b5bbcac586fc442 P 80ec10c0466246e5a4fb75ce61877a1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } }
I20260928 08:54:11.632846 32275 raft_consensus.cc:504] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } }
I20260928 08:54:11.632961 32275 leader_election.cc:288] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06 [CANDIDATE]: Term 1 election: Requested vote from peers c7fa0d4b654942aca29b1488c9dacb1b (127.23.167.193:46433), 80ec10c0466246e5a4fb75ce61877a1f (127.23.167.195:42447)
I20260928 08:54:11.632965 32276 leader_election.cc:288] T a36e512ec53b46249b5bbcac586fc442 P 80ec10c0466246e5a4fb75ce61877a1f [CANDIDATE]: Term 1 election: Requested vote from peers c7fa0d4b654942aca29b1488c9dacb1b (127.23.167.193:46433), b4a99b7bc9f54355980d72922c77af06 (127.23.167.194:33563)
I20260928 08:54:11.632963 32197 raft_consensus.cc:2379] T a36e512ec53b46249b5bbcac586fc442 P 80ec10c0466246e5a4fb75ce61877a1f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b4a99b7bc9f54355980d72922c77af06 in current term 1: Already voted for candidate 80ec10c0466246e5a4fb75ce61877a1f in this term.
I20260928 08:54:11.633131 32057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a36e512ec53b46249b5bbcac586fc442" candidate_uuid: "80ec10c0466246e5a4fb75ce61877a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a99b7bc9f54355980d72922c77af06"
I20260928 08:54:11.633199 32057 raft_consensus.cc:2379] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 80ec10c0466246e5a4fb75ce61877a1f in current term 1: Already voted for candidate b4a99b7bc9f54355980d72922c77af06 in this term.
I20260928 08:54:11.633162 32196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a36e512ec53b46249b5bbcac586fc442" candidate_uuid: "b4a99b7bc9f54355980d72922c77af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80ec10c0466246e5a4fb75ce61877a1f"
I20260928 08:54:11.633237 32196 raft_consensus.cc:2379] T a36e512ec53b46249b5bbcac586fc442 P 80ec10c0466246e5a4fb75ce61877a1f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b4a99b7bc9f54355980d72922c77af06 in current term 1: Already voted for candidate 80ec10c0466246e5a4fb75ce61877a1f in this term.
I20260928 08:54:11.633248 31917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a36e512ec53b46249b5bbcac586fc442" candidate_uuid: "b4a99b7bc9f54355980d72922c77af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7fa0d4b654942aca29b1488c9dacb1b"
I20260928 08:54:11.633307 31917 raft_consensus.cc:3037] T a36e512ec53b46249b5bbcac586fc442 P c7fa0d4b654942aca29b1488c9dacb1b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:54:11.633976 31917 raft_consensus.cc:2454] T a36e512ec53b46249b5bbcac586fc442 P c7fa0d4b654942aca29b1488c9dacb1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4a99b7bc9f54355980d72922c77af06 in term 1.
I20260928 08:54:11.634057 31916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a36e512ec53b46249b5bbcac586fc442" candidate_uuid: "80ec10c0466246e5a4fb75ce61877a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7fa0d4b654942aca29b1488c9dacb1b"
I20260928 08:54:11.634114 31990 leader_election.cc:302] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b4a99b7bc9f54355980d72922c77af06, c7fa0d4b654942aca29b1488c9dacb1b; no voters: 80ec10c0466246e5a4fb75ce61877a1f
I20260928 08:54:11.634143 31916 raft_consensus.cc:2379] T a36e512ec53b46249b5bbcac586fc442 P c7fa0d4b654942aca29b1488c9dacb1b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 80ec10c0466246e5a4fb75ce61877a1f in current term 1: Already voted for candidate b4a99b7bc9f54355980d72922c77af06 in this term.
I20260928 08:54:11.634193 32275 raft_consensus.cc:2781] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:54:11.634308 32275 raft_consensus.cc:686] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06 [term 1 LEADER]: Becoming Leader. State: Replica: b4a99b7bc9f54355980d72922c77af06, State: Running, Role: LEADER
I20260928 08:54:11.634330 32130 leader_election.cc:302] T a36e512ec53b46249b5bbcac586fc442 P 80ec10c0466246e5a4fb75ce61877a1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 80ec10c0466246e5a4fb75ce61877a1f; no voters: b4a99b7bc9f54355980d72922c77af06, c7fa0d4b654942aca29b1488c9dacb1b
I20260928 08:54:11.634375 32275 consensus_queue.cc:237] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } }
I20260928 08:54:11.634439 32276 raft_consensus.cc:2726] T a36e512ec53b46249b5bbcac586fc442 P 80ec10c0466246e5a4fb75ce61877a1f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260928 08:54:11.634672 32275 tablet_replica.cc:439] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06: TxnStatusTablet state changed. Reason: New leader b4a99b7bc9f54355980d72922c77af06. Latest consensus state: current_term: 1 leader_uuid: "b4a99b7bc9f54355980d72922c77af06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } }
I20260928 08:54:11.634745 32275 tablet_replica.cc:442] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:54:11.634897 32315 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 08:54:11.635004 32315 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 08:54:11.635109 31759 catalog_manager.cc:5495] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4a99b7bc9f54355980d72922c77af06 (127.23.167.194). New cstate: current_term: 1 leader_uuid: "b4a99b7bc9f54355980d72922c77af06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } health_report { overall_health: HEALTHY } } }
I20260928 08:54:11.643523 32267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.640746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56944 (local address 127.23.167.194:33563)
0928 08:54:11.640802 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:11.640804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:11.640811 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:54:11.640835 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:11.640836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:11.640852 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:11.640912 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:54:11.641056 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.641604 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.642732 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.642947 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.643002 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.643178 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:54:11.643283 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:54:11.643336 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:11.643417 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 08:54:11.643435 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 08:54:11.643538 32304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.640517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:33563 (local address 127.0.0.1:56944)
0928 08:54:11.640593 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:11.640606 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:11.640658 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:11.640940 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:11.640941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:11.640942 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:54:11.640997 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.641000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.641665 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.641666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.642634 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.642638 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.643008 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.643009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.643080 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.643098 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:54:11.643338 (+ 240us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:54:11.643345 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 08:54:11.643393 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 08:54:11.643456 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 08:54:11.644630 32196 raft_consensus.cc:1261] T a36e512ec53b46249b5bbcac586fc442 P 80ec10c0466246e5a4fb75ce61877a1f [term 1 FOLLOWER]: Refusing update from remote peer b4a99b7bc9f54355980d72922c77af06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:54:11.644754 31916 raft_consensus.cc:1261] T a36e512ec53b46249b5bbcac586fc442 P c7fa0d4b654942aca29b1488c9dacb1b [term 1 FOLLOWER]: Refusing update from remote peer b4a99b7bc9f54355980d72922c77af06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:54:11.644910 32275 consensus_queue.cc:1035] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:54:11.645044 32313 consensus_queue.cc:1035] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:54:11.645766 32287 tablet_replica.cc:439] T a36e512ec53b46249b5bbcac586fc442 P c7fa0d4b654942aca29b1488c9dacb1b: TxnStatusTablet state changed. Reason: New leader b4a99b7bc9f54355980d72922c77af06. Latest consensus state: current_term: 1 leader_uuid: "b4a99b7bc9f54355980d72922c77af06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } }
I20260928 08:54:11.645852 32287 tablet_replica.cc:442] T a36e512ec53b46249b5bbcac586fc442 P c7fa0d4b654942aca29b1488c9dacb1b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:54:11.645941 32276 tablet_replica.cc:439] T a36e512ec53b46249b5bbcac586fc442 P 80ec10c0466246e5a4fb75ce61877a1f: TxnStatusTablet state changed. Reason: New leader b4a99b7bc9f54355980d72922c77af06. Latest consensus state: current_term: 1 leader_uuid: "b4a99b7bc9f54355980d72922c77af06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } }
I20260928 08:54:11.646044 32276 tablet_replica.cc:442] T a36e512ec53b46249b5bbcac586fc442 P 80ec10c0466246e5a4fb75ce61877a1f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:54:11.646585 32313 tablet_replica.cc:439] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4a99b7bc9f54355980d72922c77af06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } }
I20260928 08:54:11.646677 32313 tablet_replica.cc:442] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:54:11.647608 32276 tablet_replica.cc:439] T a36e512ec53b46249b5bbcac586fc442 P 80ec10c0466246e5a4fb75ce61877a1f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4a99b7bc9f54355980d72922c77af06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } }
I20260928 08:54:11.647713 32276 tablet_replica.cc:442] T a36e512ec53b46249b5bbcac586fc442 P 80ec10c0466246e5a4fb75ce61877a1f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:54:11.652628 32267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.650449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:55477 (local address 127.23.167.194:33563)
0928 08:54:11.650494 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:11.650496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:11.650564 (+ 68us) server_negotiation.cc:407] Connection header received
0928 08:54:11.650626 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:11.650627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:11.650637 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:11.650682 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:54:11.650799 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.651273 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.652020 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.652211 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.652264 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.652418 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:54:11.652447 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:54:11.652486 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:11.652536 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 08:54:11.652560 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 08:54:11.652664 32322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.650312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:33563 (local address 127.23.167.193:55477)
0928 08:54:11.650490 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:11.650502 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:11.650569 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:11.650689 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:11.650689 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:11.650691 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:54:11.650746 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.650748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.651344 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.651345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.651946 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.651949 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.652269 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.652270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.652333 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.652350 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:54:11.652493 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:54:11.652495 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 08:54:11.652535 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 08:54:11.652590 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20260928 08:54:11.654867 32196 raft_consensus.cc:1261] T 88c9d4be3bb241a09cfb087921b75e77 P 80ec10c0466246e5a4fb75ce61877a1f [term 1 FOLLOWER]: Refusing update from remote peer c7fa0d4b654942aca29b1488c9dacb1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:54:11.654878 32057 raft_consensus.cc:1261] T 88c9d4be3bb241a09cfb087921b75e77 P b4a99b7bc9f54355980d72922c77af06 [term 1 FOLLOWER]: Refusing update from remote peer c7fa0d4b654942aca29b1488c9dacb1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:54:11.655072 32279 consensus_queue.cc:1035] T 88c9d4be3bb241a09cfb087921b75e77 P c7fa0d4b654942aca29b1488c9dacb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:54:11.655162 32287 consensus_queue.cc:1035] T 88c9d4be3bb241a09cfb087921b75e77 P c7fa0d4b654942aca29b1488c9dacb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:54:11.656703 32284 mvcc.cc:205] Tried to move back new op lower bound from 7334238829176647680 to 7334238828131016704. Current Snapshot: MvccSnapshot[applied={T|T < 7334238829176647680}]
I20260928 08:54:11.657675 32275 tablet_replica.cc:439] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4a99b7bc9f54355980d72922c77af06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } }
I20260928 08:54:11.657766 32275 tablet_replica.cc:442] T a36e512ec53b46249b5bbcac586fc442 P b4a99b7bc9f54355980d72922c77af06: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:54:11.658205 32274 tablet_replica.cc:439] T a36e512ec53b46249b5bbcac586fc442 P c7fa0d4b654942aca29b1488c9dacb1b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4a99b7bc9f54355980d72922c77af06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } }
I20260928 08:54:11.658335 32274 tablet_replica.cc:442] T a36e512ec53b46249b5bbcac586fc442 P c7fa0d4b654942aca29b1488c9dacb1b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:54:11.664871 32312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.662020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:48417 (local address 127.23.167.193:46433)
0928 08:54:11.662058 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:11.662059 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:11.662066 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:54:11.662087 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:11.662087 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:11.662097 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:11.662144 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:54:11.662276 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.662730 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.664067 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.664380 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.664419 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.664653 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:54:11.664684 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:54:11.664724 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:11.664799 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 08:54:11.664820 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 08:54:11.664952 32328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.661774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:46433 (local address 127.23.167.194:48417)
0928 08:54:11.661947 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:11.661959 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:11.662010 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:11.662149 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:11.662150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:11.662152 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:54:11.662222 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.662223 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.662787 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.662788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.663973 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.663975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.664493 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.664494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.664561 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.664578 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:54:11.664747 (+ 169us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:54:11.664748 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 08:54:11.664797 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 08:54:11.664842 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
I20260928 08:54:11.673183 31759 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:54:11.673273 31759 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:54:11.673926 31759 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc1b95ca7c2d48d5ae1c2c1064bcb2b1: Sending DeleteTablet for 3 replicas of tablet 88c9d4be3bb241a09cfb087921b75e77
I20260928 08:54:11.674165 31897 tablet_service.cc:1514] Processing DeleteTablet for tablet 88c9d4be3bb241a09cfb087921b75e77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:54:11 UTC) from {username='oryx', principal='oryx/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:37296
I20260928 08:54:11.674194 32037 tablet_service.cc:1514] Processing DeleteTablet for tablet 88c9d4be3bb241a09cfb087921b75e77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:54:11 UTC) from {username='oryx', principal='oryx/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:56940
I20260928 08:54:11.674269 32294 tablet_replica.cc:330] T 88c9d4be3bb241a09cfb087921b75e77 P c7fa0d4b654942aca29b1488c9dacb1b: stopping tablet replica
I20260928 08:54:11.674278 32292 tablet_replica.cc:330] T 88c9d4be3bb241a09cfb087921b75e77 P b4a99b7bc9f54355980d72922c77af06: stopping tablet replica
I20260928 08:54:11.674326 32292 raft_consensus.cc:2229] T 88c9d4be3bb241a09cfb087921b75e77 P b4a99b7bc9f54355980d72922c77af06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:54:11.674329 32294 raft_consensus.cc:2229] T 88c9d4be3bb241a09cfb087921b75e77 P c7fa0d4b654942aca29b1488c9dacb1b [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:54:11.674355 32292 raft_consensus.cc:2258] T 88c9d4be3bb241a09cfb087921b75e77 P b4a99b7bc9f54355980d72922c77af06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:54:11.674377 32294 raft_consensus.cc:2258] T 88c9d4be3bb241a09cfb087921b75e77 P c7fa0d4b654942aca29b1488c9dacb1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:54:11.674361 32177 tablet_service.cc:1514] Processing DeleteTablet for tablet 88c9d4be3bb241a09cfb087921b75e77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:54:11 UTC) from {username='oryx', principal='oryx/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:46584
I20260928 08:54:11.674454 32293 tablet_replica.cc:330] T 88c9d4be3bb241a09cfb087921b75e77 P 80ec10c0466246e5a4fb75ce61877a1f: stopping tablet replica
I20260928 08:54:11.674499 32293 raft_consensus.cc:2229] T 88c9d4be3bb241a09cfb087921b75e77 P 80ec10c0466246e5a4fb75ce61877a1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:54:11.674530 32293 raft_consensus.cc:2258] T 88c9d4be3bb241a09cfb087921b75e77 P 80ec10c0466246e5a4fb75ce61877a1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:54:11.674698 32294 ts_tablet_manager.cc:1905] T 88c9d4be3bb241a09cfb087921b75e77 P c7fa0d4b654942aca29b1488c9dacb1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:54:11.674736 32292 ts_tablet_manager.cc:1905] T 88c9d4be3bb241a09cfb087921b75e77 P b4a99b7bc9f54355980d72922c77af06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:54:11.674808 32293 ts_tablet_manager.cc:1905] T 88c9d4be3bb241a09cfb087921b75e77 P 80ec10c0466246e5a4fb75ce61877a1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:54:11.676626 32292 ts_tablet_manager.cc:1918] T 88c9d4be3bb241a09cfb087921b75e77 P b4a99b7bc9f54355980d72922c77af06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:54:11.676636 32293 ts_tablet_manager.cc:1918] T 88c9d4be3bb241a09cfb087921b75e77 P 80ec10c0466246e5a4fb75ce61877a1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:54:11.676680 32292 log.cc:1196] T 88c9d4be3bb241a09cfb087921b75e77 P b4a99b7bc9f54355980d72922c77af06: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-1/wal/wals/88c9d4be3bb241a09cfb087921b75e77
I20260928 08:54:11.676688 32293 log.cc:1196] T 88c9d4be3bb241a09cfb087921b75e77 P 80ec10c0466246e5a4fb75ce61877a1f: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-2/wal/wals/88c9d4be3bb241a09cfb087921b75e77
I20260928 08:54:11.676865 32292 ts_tablet_manager.cc:1939] T 88c9d4be3bb241a09cfb087921b75e77 P b4a99b7bc9f54355980d72922c77af06: Deleting consensus metadata
I20260928 08:54:11.676887 32293 ts_tablet_manager.cc:1939] T 88c9d4be3bb241a09cfb087921b75e77 P 80ec10c0466246e5a4fb75ce61877a1f: Deleting consensus metadata
I20260928 08:54:11.677184 31744 catalog_manager.cc:4841] TS 80ec10c0466246e5a4fb75ce61877a1f (127.23.167.195:42447): tablet 88c9d4be3bb241a09cfb087921b75e77 (table test-table [id=dcf39fccf00545279173f9df153a6930]) successfully deleted
I20260928 08:54:11.678187 31745 catalog_manager.cc:4841] TS b4a99b7bc9f54355980d72922c77af06 (127.23.167.194:33563): tablet 88c9d4be3bb241a09cfb087921b75e77 (table test-table [id=dcf39fccf00545279173f9df153a6930]) successfully deleted
I20260928 08:54:11.679177 32294 ts_tablet_manager.cc:1918] T 88c9d4be3bb241a09cfb087921b75e77 P c7fa0d4b654942aca29b1488c9dacb1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:54:11.679225 32294 log.cc:1196] T 88c9d4be3bb241a09cfb087921b75e77 P c7fa0d4b654942aca29b1488c9dacb1b: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-0/wal/wals/88c9d4be3bb241a09cfb087921b75e77
I20260928 08:54:11.679414 32294 ts_tablet_manager.cc:1939] T 88c9d4be3bb241a09cfb087921b75e77 P c7fa0d4b654942aca29b1488c9dacb1b: Deleting consensus metadata
I20260928 08:54:11.679667 31746 catalog_manager.cc:4841] TS c7fa0d4b654942aca29b1488c9dacb1b (127.23.167.193:46433): tablet 88c9d4be3bb241a09cfb087921b75e77 (table test-table [id=dcf39fccf00545279173f9df153a6930]) successfully deleted
Sep 28 08:54:11 dist-test-slave-pbjp krb5kdc[31715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 08:54:11 dist-test-slave-pbjp krb5kdc[31715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:11 dist-test-slave-pbjp krb5kdc[31715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 08:54:11.704097 31810 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:11.698473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35428 (local address 127.23.167.254:42661)
0928 08:54:11.698538 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:11.698540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:11.698569 (+ 29us) server_negotiation.cc:407] Connection header received
0928 08:54:11.698627 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:11.698628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:11.698639 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:11.698711 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:11.699241 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.700013 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.700766 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.701098 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.701168 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.704028 (+ 2860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35428: BlockingRecv error: recv got EOF from 127.0.0.1:35428 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 28 08:54:11 dist-test-slave-pbjp krb5kdc[31715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM
I20260928 08:54:11.712072 31810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.704993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35438 (local address 127.23.167.254:42661)
0928 08:54:11.705045 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:11.705047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:11.705137 (+ 90us) server_negotiation.cc:407] Connection header received
0928 08:54:11.705190 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:11.705191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:11.705202 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:11.705261 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:11.705686 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.706394 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.707078 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.707363 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.707432 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.709867 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:54:11.709868 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:54:11.709868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:54:11.709875 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:54:11.711084 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:11.711455 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:11.711456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:11.711456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:11.711501 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:11.711665 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:11.711665 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:11.711665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:11.711755 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:54:11.711824 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:11.711953 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 08:54:11.712004 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 08:54:11.713552 31759 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:54:11.713711 31759 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:54:11.715112 31897 tablet_service.cc:1467] Processing CreateTablet for tablet 7da5a118f34546ac81c4c50b61cc0828 (DEFAULT_TABLE table=test-table [id=8e137c12bab84bfc8e1255553d64371c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:54:11.715245 31897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7da5a118f34546ac81c4c50b61cc0828. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:54:11.715463 32177 tablet_service.cc:1467] Processing CreateTablet for tablet 7da5a118f34546ac81c4c50b61cc0828 (DEFAULT_TABLE table=test-table [id=8e137c12bab84bfc8e1255553d64371c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:54:11.715569 32177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7da5a118f34546ac81c4c50b61cc0828. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:54:11.716548 32268 tablet_bootstrap.cc:492] T 7da5a118f34546ac81c4c50b61cc0828 P c7fa0d4b654942aca29b1488c9dacb1b: Bootstrap starting.
I20260928 08:54:11.716840 32268 tablet_bootstrap.cc:654] T 7da5a118f34546ac81c4c50b61cc0828 P c7fa0d4b654942aca29b1488c9dacb1b: Neither blocks nor log segments found. Creating new log.
I20260928 08:54:11.716817 32037 tablet_service.cc:1467] Processing CreateTablet for tablet 7da5a118f34546ac81c4c50b61cc0828 (DEFAULT_TABLE table=test-table [id=8e137c12bab84bfc8e1255553d64371c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:54:11.716926 32037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7da5a118f34546ac81c4c50b61cc0828. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:54:11.717909 32270 tablet_bootstrap.cc:492] T 7da5a118f34546ac81c4c50b61cc0828 P b4a99b7bc9f54355980d72922c77af06: Bootstrap starting.
I20260928 08:54:11.717923 32269 tablet_bootstrap.cc:492] T 7da5a118f34546ac81c4c50b61cc0828 P 80ec10c0466246e5a4fb75ce61877a1f: Bootstrap starting.
I20260928 08:54:11.718338 32270 tablet_bootstrap.cc:654] T 7da5a118f34546ac81c4c50b61cc0828 P b4a99b7bc9f54355980d72922c77af06: Neither blocks nor log segments found. Creating new log.
I20260928 08:54:11.718510 32269 tablet_bootstrap.cc:654] T 7da5a118f34546ac81c4c50b61cc0828 P 80ec10c0466246e5a4fb75ce61877a1f: Neither blocks nor log segments found. Creating new log.
I20260928 08:54:11.718957 32270 tablet_bootstrap.cc:492] T 7da5a118f34546ac81c4c50b61cc0828 P b4a99b7bc9f54355980d72922c77af06: No bootstrap required, opened a new log
I20260928 08:54:11.719003 32270 ts_tablet_manager.cc:1397] T 7da5a118f34546ac81c4c50b61cc0828 P b4a99b7bc9f54355980d72922c77af06: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 08:54:11.719174 32270 raft_consensus.cc:348] T 7da5a118f34546ac81c4c50b61cc0828 P b4a99b7bc9f54355980d72922c77af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } }
I20260928 08:54:11.719264 32270 raft_consensus.cc:374] T 7da5a118f34546ac81c4c50b61cc0828 P b4a99b7bc9f54355980d72922c77af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:54:11.719312 32270 raft_consensus.cc:729] T 7da5a118f34546ac81c4c50b61cc0828 P b4a99b7bc9f54355980d72922c77af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4a99b7bc9f54355980d72922c77af06, State: Initialized, Role: FOLLOWER
I20260928 08:54:11.719357 32270 consensus_queue.cc:260] T 7da5a118f34546ac81c4c50b61cc0828 P b4a99b7bc9f54355980d72922c77af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } }
I20260928 08:54:11.719456 32270 ts_tablet_manager.cc:1428] T 7da5a118f34546ac81c4c50b61cc0828 P b4a99b7bc9f54355980d72922c77af06: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 08:54:11.719574 32269 tablet_bootstrap.cc:492] T 7da5a118f34546ac81c4c50b61cc0828 P 80ec10c0466246e5a4fb75ce61877a1f: No bootstrap required, opened a new log
I20260928 08:54:11.719614 32269 ts_tablet_manager.cc:1397] T 7da5a118f34546ac81c4c50b61cc0828 P 80ec10c0466246e5a4fb75ce61877a1f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:54:11.719761 32269 raft_consensus.cc:348] T 7da5a118f34546ac81c4c50b61cc0828 P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } }
I20260928 08:54:11.719816 32269 raft_consensus.cc:374] T 7da5a118f34546ac81c4c50b61cc0828 P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:54:11.719832 32269 raft_consensus.cc:729] T 7da5a118f34546ac81c4c50b61cc0828 P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80ec10c0466246e5a4fb75ce61877a1f, State: Initialized, Role: FOLLOWER
I20260928 08:54:11.719868 32269 consensus_queue.cc:260] T 7da5a118f34546ac81c4c50b61cc0828 P 80ec10c0466246e5a4fb75ce61877a1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } }
I20260928 08:54:11.719954 32269 ts_tablet_manager.cc:1428] T 7da5a118f34546ac81c4c50b61cc0828 P 80ec10c0466246e5a4fb75ce61877a1f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:11.720382 32268 tablet_bootstrap.cc:492] T 7da5a118f34546ac81c4c50b61cc0828 P c7fa0d4b654942aca29b1488c9dacb1b: No bootstrap required, opened a new log
I20260928 08:54:11.720458 32268 ts_tablet_manager.cc:1397] T 7da5a118f34546ac81c4c50b61cc0828 P c7fa0d4b654942aca29b1488c9dacb1b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 08:54:11.720566 32268 raft_consensus.cc:348] T 7da5a118f34546ac81c4c50b61cc0828 P c7fa0d4b654942aca29b1488c9dacb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } }
I20260928 08:54:11.720625 32268 raft_consensus.cc:374] T 7da5a118f34546ac81c4c50b61cc0828 P c7fa0d4b654942aca29b1488c9dacb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:54:11.720643 32268 raft_consensus.cc:729] T 7da5a118f34546ac81c4c50b61cc0828 P c7fa0d4b654942aca29b1488c9dacb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7fa0d4b654942aca29b1488c9dacb1b, State: Initialized, Role: FOLLOWER
I20260928 08:54:11.720680 32268 consensus_queue.cc:260] T 7da5a118f34546ac81c4c50b61cc0828 P c7fa0d4b654942aca29b1488c9dacb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } }
I20260928 08:54:11.720783 32268 ts_tablet_manager.cc:1428] T 7da5a118f34546ac81c4c50b61cc0828 P c7fa0d4b654942aca29b1488c9dacb1b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:11.778285 32276 raft_consensus.cc:482] T 7da5a118f34546ac81c4c50b61cc0828 P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:54:11.778367 32276 raft_consensus.cc:504] T 7da5a118f34546ac81c4c50b61cc0828 P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } }
I20260928 08:54:11.778558 32276 leader_election.cc:288] T 7da5a118f34546ac81c4c50b61cc0828 P 80ec10c0466246e5a4fb75ce61877a1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7fa0d4b654942aca29b1488c9dacb1b (127.23.167.193:46433), b4a99b7bc9f54355980d72922c77af06 (127.23.167.194:33563)
I20260928 08:54:11.778832 32057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da5a118f34546ac81c4c50b61cc0828" candidate_uuid: "80ec10c0466246e5a4fb75ce61877a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a99b7bc9f54355980d72922c77af06" is_pre_election: true
I20260928 08:54:11.778862 31916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da5a118f34546ac81c4c50b61cc0828" candidate_uuid: "80ec10c0466246e5a4fb75ce61877a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" is_pre_election: true
I20260928 08:54:11.778972 32057 raft_consensus.cc:2454] T 7da5a118f34546ac81c4c50b61cc0828 P b4a99b7bc9f54355980d72922c77af06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80ec10c0466246e5a4fb75ce61877a1f in term 0.
I20260928 08:54:11.778971 31916 raft_consensus.cc:2454] T 7da5a118f34546ac81c4c50b61cc0828 P c7fa0d4b654942aca29b1488c9dacb1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80ec10c0466246e5a4fb75ce61877a1f in term 0.
I20260928 08:54:11.779115 32130 leader_election.cc:302] T 7da5a118f34546ac81c4c50b61cc0828 P 80ec10c0466246e5a4fb75ce61877a1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80ec10c0466246e5a4fb75ce61877a1f, c7fa0d4b654942aca29b1488c9dacb1b; no voters:
I20260928 08:54:11.779212 32276 raft_consensus.cc:2781] T 7da5a118f34546ac81c4c50b61cc0828 P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:54:11.779253 32276 raft_consensus.cc:482] T 7da5a118f34546ac81c4c50b61cc0828 P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:54:11.779271 32276 raft_consensus.cc:3037] T 7da5a118f34546ac81c4c50b61cc0828 P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:54:11.779953 32276 raft_consensus.cc:504] T 7da5a118f34546ac81c4c50b61cc0828 P 80ec10c0466246e5a4fb75ce61877a1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } }
I20260928 08:54:11.780066 32276 leader_election.cc:288] T 7da5a118f34546ac81c4c50b61cc0828 P 80ec10c0466246e5a4fb75ce61877a1f [CANDIDATE]: Term 1 election: Requested vote from peers c7fa0d4b654942aca29b1488c9dacb1b (127.23.167.193:46433), b4a99b7bc9f54355980d72922c77af06 (127.23.167.194:33563)
I20260928 08:54:11.780211 32057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da5a118f34546ac81c4c50b61cc0828" candidate_uuid: "80ec10c0466246e5a4fb75ce61877a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a99b7bc9f54355980d72922c77af06"
I20260928 08:54:11.780275 32057 raft_consensus.cc:3037] T 7da5a118f34546ac81c4c50b61cc0828 P b4a99b7bc9f54355980d72922c77af06 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:54:11.780220 31916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da5a118f34546ac81c4c50b61cc0828" candidate_uuid: "80ec10c0466246e5a4fb75ce61877a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7fa0d4b654942aca29b1488c9dacb1b"
I20260928 08:54:11.780304 31916 raft_consensus.cc:3037] T 7da5a118f34546ac81c4c50b61cc0828 P c7fa0d4b654942aca29b1488c9dacb1b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:54:11.780792 32057 raft_consensus.cc:2454] T 7da5a118f34546ac81c4c50b61cc0828 P b4a99b7bc9f54355980d72922c77af06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80ec10c0466246e5a4fb75ce61877a1f in term 1.
I20260928 08:54:11.780963 32129 leader_election.cc:302] T 7da5a118f34546ac81c4c50b61cc0828 P 80ec10c0466246e5a4fb75ce61877a1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80ec10c0466246e5a4fb75ce61877a1f, b4a99b7bc9f54355980d72922c77af06; no voters:
I20260928 08:54:11.781006 31916 raft_consensus.cc:2454] T 7da5a118f34546ac81c4c50b61cc0828 P c7fa0d4b654942aca29b1488c9dacb1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80ec10c0466246e5a4fb75ce61877a1f in term 1.
I20260928 08:54:11.781103 32276 raft_consensus.cc:2781] T 7da5a118f34546ac81c4c50b61cc0828 P 80ec10c0466246e5a4fb75ce61877a1f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:54:11.781237 32276 raft_consensus.cc:686] T 7da5a118f34546ac81c4c50b61cc0828 P 80ec10c0466246e5a4fb75ce61877a1f [term 1 LEADER]: Becoming Leader. State: Replica: 80ec10c0466246e5a4fb75ce61877a1f, State: Running, Role: LEADER
I20260928 08:54:11.781315 32276 consensus_queue.cc:237] T 7da5a118f34546ac81c4c50b61cc0828 P 80ec10c0466246e5a4fb75ce61877a1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } }
I20260928 08:54:11.781999 31759 catalog_manager.cc:5495] T 7da5a118f34546ac81c4c50b61cc0828 P 80ec10c0466246e5a4fb75ce61877a1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80ec10c0466246e5a4fb75ce61877a1f (127.23.167.195). New cstate: current_term: 1 leader_uuid: "80ec10c0466246e5a4fb75ce61877a1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } health_report { overall_health: HEALTHY } } }
I20260928 08:54:11.796703 32265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.794675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46588 (local address 127.23.167.195:42447)
0928 08:54:11.794727 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:11.794728 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:11.794735 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:54:11.794767 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:11.794768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:11.794784 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:11.794844 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:54:11.794935 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.795393 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.796090 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.796275 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.796316 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.796416 (+ 100us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:54:11.796522 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:54:11.796571 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:11.796601 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 08:54:11.796614 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 08:54:11.797626 32057 raft_consensus.cc:1261] T 7da5a118f34546ac81c4c50b61cc0828 P b4a99b7bc9f54355980d72922c77af06 [term 1 FOLLOWER]: Refusing update from remote peer 80ec10c0466246e5a4fb75ce61877a1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:54:11.797776 31916 raft_consensus.cc:1261] T 7da5a118f34546ac81c4c50b61cc0828 P c7fa0d4b654942aca29b1488c9dacb1b [term 1 FOLLOWER]: Refusing update from remote peer 80ec10c0466246e5a4fb75ce61877a1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:54:11.797863 32345 consensus_queue.cc:1035] T 7da5a118f34546ac81c4c50b61cc0828 P 80ec10c0466246e5a4fb75ce61877a1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:54:11.798012 32276 consensus_queue.cc:1035] T 7da5a118f34546ac81c4c50b61cc0828 P 80ec10c0466246e5a4fb75ce61877a1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:54:11.801116 31760 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:54:11.801205 31760 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:54:11.801833 31760 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc1b95ca7c2d48d5ae1c2c1064bcb2b1: Sending DeleteTablet for 3 replicas of tablet 7da5a118f34546ac81c4c50b61cc0828
I20260928 08:54:11.802075 32037 tablet_service.cc:1514] Processing DeleteTablet for tablet 7da5a118f34546ac81c4c50b61cc0828 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:54:11 UTC) from {username='oryx', principal='oryx/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:56940
I20260928 08:54:11.802156 32177 tablet_service.cc:1514] Processing DeleteTablet for tablet 7da5a118f34546ac81c4c50b61cc0828 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:54:11 UTC) from {username='oryx', principal='oryx/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:46584
I20260928 08:54:11.802173 31897 tablet_service.cc:1514] Processing DeleteTablet for tablet 7da5a118f34546ac81c4c50b61cc0828 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:54:11 UTC) from {username='oryx', principal='oryx/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:37296
I20260928 08:54:11.802246 32293 tablet_replica.cc:330] T 7da5a118f34546ac81c4c50b61cc0828 P 80ec10c0466246e5a4fb75ce61877a1f: stopping tablet replica
I20260928 08:54:11.802253 32292 tablet_replica.cc:330] T 7da5a118f34546ac81c4c50b61cc0828 P b4a99b7bc9f54355980d72922c77af06: stopping tablet replica
I20260928 08:54:11.802297 32293 raft_consensus.cc:2229] T 7da5a118f34546ac81c4c50b61cc0828 P 80ec10c0466246e5a4fb75ce61877a1f [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:54:11.802299 32292 raft_consensus.cc:2229] T 7da5a118f34546ac81c4c50b61cc0828 P b4a99b7bc9f54355980d72922c77af06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:54:11.802342 32292 raft_consensus.cc:2258] T 7da5a118f34546ac81c4c50b61cc0828 P b4a99b7bc9f54355980d72922c77af06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:54:11.802359 32293 raft_consensus.cc:2258] T 7da5a118f34546ac81c4c50b61cc0828 P 80ec10c0466246e5a4fb75ce61877a1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:54:11.802402 32294 tablet_replica.cc:330] T 7da5a118f34546ac81c4c50b61cc0828 P c7fa0d4b654942aca29b1488c9dacb1b: stopping tablet replica
I20260928 08:54:11.802448 32294 raft_consensus.cc:2229] T 7da5a118f34546ac81c4c50b61cc0828 P c7fa0d4b654942aca29b1488c9dacb1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:54:11.802502 32294 raft_consensus.cc:2258] T 7da5a118f34546ac81c4c50b61cc0828 P c7fa0d4b654942aca29b1488c9dacb1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:54:11.802570 31760 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:54:11.802702 31760 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:54:11.802726 32293 ts_tablet_manager.cc:1905] T 7da5a118f34546ac81c4c50b61cc0828 P 80ec10c0466246e5a4fb75ce61877a1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:54:11.803129 32294 ts_tablet_manager.cc:1905] T 7da5a118f34546ac81c4c50b61cc0828 P c7fa0d4b654942aca29b1488c9dacb1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:54:11.804297 32293 ts_tablet_manager.cc:1918] T 7da5a118f34546ac81c4c50b61cc0828 P 80ec10c0466246e5a4fb75ce61877a1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:54:11.804346 32293 log.cc:1196] T 7da5a118f34546ac81c4c50b61cc0828 P 80ec10c0466246e5a4fb75ce61877a1f: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-2/wal/wals/7da5a118f34546ac81c4c50b61cc0828
I20260928 08:54:11.804538 32293 ts_tablet_manager.cc:1939] T 7da5a118f34546ac81c4c50b61cc0828 P 80ec10c0466246e5a4fb75ce61877a1f: Deleting consensus metadata
I20260928 08:54:11.804540 32294 ts_tablet_manager.cc:1918] T 7da5a118f34546ac81c4c50b61cc0828 P c7fa0d4b654942aca29b1488c9dacb1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:54:11.804580 32294 log.cc:1196] T 7da5a118f34546ac81c4c50b61cc0828 P c7fa0d4b654942aca29b1488c9dacb1b: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-0/wal/wals/7da5a118f34546ac81c4c50b61cc0828
I20260928 08:54:11.804744 32294 ts_tablet_manager.cc:1939] T 7da5a118f34546ac81c4c50b61cc0828 P c7fa0d4b654942aca29b1488c9dacb1b: Deleting consensus metadata
I20260928 08:54:11.805159 31746 catalog_manager.cc:4841] TS c7fa0d4b654942aca29b1488c9dacb1b (127.23.167.193:46433): tablet 7da5a118f34546ac81c4c50b61cc0828 (table test-table [id=8e137c12bab84bfc8e1255553d64371c]) successfully deleted
I20260928 08:54:11.805176 32292 ts_tablet_manager.cc:1905] T 7da5a118f34546ac81c4c50b61cc0828 P b4a99b7bc9f54355980d72922c77af06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:54:11.805424 31897 tablet_service.cc:1467] Processing CreateTablet for tablet c76e757c44aa42008cf59d51027e0a3c (DEFAULT_TABLE table=test-table [id=ba9159ca74dc496da425457990780dfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:54:11.805513 31897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c76e757c44aa42008cf59d51027e0a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:54:11.806545 32268 tablet_bootstrap.cc:492] T c76e757c44aa42008cf59d51027e0a3c P c7fa0d4b654942aca29b1488c9dacb1b: Bootstrap starting.
I20260928 08:54:11.806551 32292 ts_tablet_manager.cc:1918] T 7da5a118f34546ac81c4c50b61cc0828 P b4a99b7bc9f54355980d72922c77af06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:54:11.806598 32292 log.cc:1196] T 7da5a118f34546ac81c4c50b61cc0828 P b4a99b7bc9f54355980d72922c77af06: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-1/wal/wals/7da5a118f34546ac81c4c50b61cc0828
I20260928 08:54:11.806699 31744 catalog_manager.cc:4841] TS 80ec10c0466246e5a4fb75ce61877a1f (127.23.167.195:42447): tablet 7da5a118f34546ac81c4c50b61cc0828 (table test-table [id=8e137c12bab84bfc8e1255553d64371c]) successfully deleted
I20260928 08:54:11.806783 32292 ts_tablet_manager.cc:1939] T 7da5a118f34546ac81c4c50b61cc0828 P b4a99b7bc9f54355980d72922c77af06: Deleting consensus metadata
I20260928 08:54:11.806854 32268 tablet_bootstrap.cc:654] T c76e757c44aa42008cf59d51027e0a3c P c7fa0d4b654942aca29b1488c9dacb1b: Neither blocks nor log segments found. Creating new log.
I20260928 08:54:11.806919 32037 tablet_service.cc:1467] Processing CreateTablet for tablet c76e757c44aa42008cf59d51027e0a3c (DEFAULT_TABLE table=test-table [id=ba9159ca74dc496da425457990780dfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:54:11.806948 32177 tablet_service.cc:1467] Processing CreateTablet for tablet c76e757c44aa42008cf59d51027e0a3c (DEFAULT_TABLE table=test-table [id=ba9159ca74dc496da425457990780dfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:54:11.806999 32037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c76e757c44aa42008cf59d51027e0a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:54:11.807037 32177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c76e757c44aa42008cf59d51027e0a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:54:11.808063 32270 tablet_bootstrap.cc:492] T c76e757c44aa42008cf59d51027e0a3c P b4a99b7bc9f54355980d72922c77af06: Bootstrap starting.
I20260928 08:54:11.808075 32269 tablet_bootstrap.cc:492] T c76e757c44aa42008cf59d51027e0a3c P 80ec10c0466246e5a4fb75ce61877a1f: Bootstrap starting.
I20260928 08:54:11.808350 32270 tablet_bootstrap.cc:654] T c76e757c44aa42008cf59d51027e0a3c P b4a99b7bc9f54355980d72922c77af06: Neither blocks nor log segments found. Creating new log.
I20260928 08:54:11.808446 32269 tablet_bootstrap.cc:654] T c76e757c44aa42008cf59d51027e0a3c P 80ec10c0466246e5a4fb75ce61877a1f: Neither blocks nor log segments found. Creating new log.
I20260928 08:54:11.808821 32270 tablet_bootstrap.cc:492] T c76e757c44aa42008cf59d51027e0a3c P b4a99b7bc9f54355980d72922c77af06: No bootstrap required, opened a new log
I20260928 08:54:11.808874 32270 ts_tablet_manager.cc:1397] T c76e757c44aa42008cf59d51027e0a3c P b4a99b7bc9f54355980d72922c77af06: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 08:54:11.808916 31745 catalog_manager.cc:4841] TS b4a99b7bc9f54355980d72922c77af06 (127.23.167.194:33563): tablet 7da5a118f34546ac81c4c50b61cc0828 (table test-table [id=8e137c12bab84bfc8e1255553d64371c]) successfully deleted
I20260928 08:54:11.809103 32270 raft_consensus.cc:348] T c76e757c44aa42008cf59d51027e0a3c P b4a99b7bc9f54355980d72922c77af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } }
I20260928 08:54:11.809229 32270 raft_consensus.cc:374] T c76e757c44aa42008cf59d51027e0a3c P b4a99b7bc9f54355980d72922c77af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:54:11.809255 32270 raft_consensus.cc:729] T c76e757c44aa42008cf59d51027e0a3c P b4a99b7bc9f54355980d72922c77af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4a99b7bc9f54355980d72922c77af06, State: Initialized, Role: FOLLOWER
I20260928 08:54:11.809299 32270 consensus_queue.cc:260] T c76e757c44aa42008cf59d51027e0a3c P b4a99b7bc9f54355980d72922c77af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } }
I20260928 08:54:11.809392 32270 ts_tablet_manager.cc:1428] T c76e757c44aa42008cf59d51027e0a3c P b4a99b7bc9f54355980d72922c77af06: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:11.809746 32269 tablet_bootstrap.cc:492] T c76e757c44aa42008cf59d51027e0a3c P 80ec10c0466246e5a4fb75ce61877a1f: No bootstrap required, opened a new log
I20260928 08:54:11.809792 32269 ts_tablet_manager.cc:1397] T c76e757c44aa42008cf59d51027e0a3c P 80ec10c0466246e5a4fb75ce61877a1f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:54:11.809811 32268 tablet_bootstrap.cc:492] T c76e757c44aa42008cf59d51027e0a3c P c7fa0d4b654942aca29b1488c9dacb1b: No bootstrap required, opened a new log
I20260928 08:54:11.809851 32268 ts_tablet_manager.cc:1397] T c76e757c44aa42008cf59d51027e0a3c P c7fa0d4b654942aca29b1488c9dacb1b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 08:54:11.809933 32269 raft_consensus.cc:348] T c76e757c44aa42008cf59d51027e0a3c P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } }
I20260928 08:54:11.809996 32269 raft_consensus.cc:374] T c76e757c44aa42008cf59d51027e0a3c P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:54:11.809947 32268 raft_consensus.cc:348] T c76e757c44aa42008cf59d51027e0a3c P c7fa0d4b654942aca29b1488c9dacb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } }
I20260928 08:54:11.810016 32269 raft_consensus.cc:729] T c76e757c44aa42008cf59d51027e0a3c P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80ec10c0466246e5a4fb75ce61877a1f, State: Initialized, Role: FOLLOWER
I20260928 08:54:11.810019 32268 raft_consensus.cc:374] T c76e757c44aa42008cf59d51027e0a3c P c7fa0d4b654942aca29b1488c9dacb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:54:11.810038 32268 raft_consensus.cc:729] T c76e757c44aa42008cf59d51027e0a3c P c7fa0d4b654942aca29b1488c9dacb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7fa0d4b654942aca29b1488c9dacb1b, State: Initialized, Role: FOLLOWER
I20260928 08:54:11.810051 32269 consensus_queue.cc:260] T c76e757c44aa42008cf59d51027e0a3c P 80ec10c0466246e5a4fb75ce61877a1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } }
I20260928 08:54:11.810077 32268 consensus_queue.cc:260] T c76e757c44aa42008cf59d51027e0a3c P c7fa0d4b654942aca29b1488c9dacb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } }
I20260928 08:54:11.810145 32269 ts_tablet_manager.cc:1428] T c76e757c44aa42008cf59d51027e0a3c P 80ec10c0466246e5a4fb75ce61877a1f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:11.810158 32268 ts_tablet_manager.cc:1428] T c76e757c44aa42008cf59d51027e0a3c P c7fa0d4b654942aca29b1488c9dacb1b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:11.901682 32350 raft_consensus.cc:482] T c76e757c44aa42008cf59d51027e0a3c P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:54:11.901779 32350 raft_consensus.cc:504] T c76e757c44aa42008cf59d51027e0a3c P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } }
I20260928 08:54:11.901973 32350 leader_election.cc:288] T c76e757c44aa42008cf59d51027e0a3c P 80ec10c0466246e5a4fb75ce61877a1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7fa0d4b654942aca29b1488c9dacb1b (127.23.167.193:46433), b4a99b7bc9f54355980d72922c77af06 (127.23.167.194:33563)
I20260928 08:54:11.902168 31916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c76e757c44aa42008cf59d51027e0a3c" candidate_uuid: "80ec10c0466246e5a4fb75ce61877a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" is_pre_election: true
I20260928 08:54:11.902283 31916 raft_consensus.cc:2454] T c76e757c44aa42008cf59d51027e0a3c P c7fa0d4b654942aca29b1488c9dacb1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80ec10c0466246e5a4fb75ce61877a1f in term 0.
I20260928 08:54:11.902415 32130 leader_election.cc:302] T c76e757c44aa42008cf59d51027e0a3c P 80ec10c0466246e5a4fb75ce61877a1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80ec10c0466246e5a4fb75ce61877a1f, c7fa0d4b654942aca29b1488c9dacb1b; no voters:
I20260928 08:54:11.902400 32057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c76e757c44aa42008cf59d51027e0a3c" candidate_uuid: "80ec10c0466246e5a4fb75ce61877a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a99b7bc9f54355980d72922c77af06" is_pre_election: true
I20260928 08:54:11.902484 32057 raft_consensus.cc:2454] T c76e757c44aa42008cf59d51027e0a3c P b4a99b7bc9f54355980d72922c77af06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80ec10c0466246e5a4fb75ce61877a1f in term 0.
I20260928 08:54:11.902509 32350 raft_consensus.cc:2781] T c76e757c44aa42008cf59d51027e0a3c P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:54:11.902535 32350 raft_consensus.cc:482] T c76e757c44aa42008cf59d51027e0a3c P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:54:11.902556 32350 raft_consensus.cc:3037] T c76e757c44aa42008cf59d51027e0a3c P 80ec10c0466246e5a4fb75ce61877a1f [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:54:11.903270 32350 raft_consensus.cc:504] T c76e757c44aa42008cf59d51027e0a3c P 80ec10c0466246e5a4fb75ce61877a1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } }
I20260928 08:54:11.903384 32350 leader_election.cc:288] T c76e757c44aa42008cf59d51027e0a3c P 80ec10c0466246e5a4fb75ce61877a1f [CANDIDATE]: Term 1 election: Requested vote from peers c7fa0d4b654942aca29b1488c9dacb1b (127.23.167.193:46433), b4a99b7bc9f54355980d72922c77af06 (127.23.167.194:33563)
I20260928 08:54:11.903580 31916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c76e757c44aa42008cf59d51027e0a3c" candidate_uuid: "80ec10c0466246e5a4fb75ce61877a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7fa0d4b654942aca29b1488c9dacb1b"
I20260928 08:54:11.903723 31916 raft_consensus.cc:3037] T c76e757c44aa42008cf59d51027e0a3c P c7fa0d4b654942aca29b1488c9dacb1b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:54:11.903698 32057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c76e757c44aa42008cf59d51027e0a3c" candidate_uuid: "80ec10c0466246e5a4fb75ce61877a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a99b7bc9f54355980d72922c77af06"
I20260928 08:54:11.903820 32057 raft_consensus.cc:3037] T c76e757c44aa42008cf59d51027e0a3c P b4a99b7bc9f54355980d72922c77af06 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:54:11.904584 32057 raft_consensus.cc:2454] T c76e757c44aa42008cf59d51027e0a3c P b4a99b7bc9f54355980d72922c77af06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80ec10c0466246e5a4fb75ce61877a1f in term 1.
I20260928 08:54:11.904726 31916 raft_consensus.cc:2454] T c76e757c44aa42008cf59d51027e0a3c P c7fa0d4b654942aca29b1488c9dacb1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80ec10c0466246e5a4fb75ce61877a1f in term 1.
I20260928 08:54:11.904747 32129 leader_election.cc:302] T c76e757c44aa42008cf59d51027e0a3c P 80ec10c0466246e5a4fb75ce61877a1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80ec10c0466246e5a4fb75ce61877a1f, b4a99b7bc9f54355980d72922c77af06; no voters:
I20260928 08:54:11.904857 32350 raft_consensus.cc:2781] T c76e757c44aa42008cf59d51027e0a3c P 80ec10c0466246e5a4fb75ce61877a1f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:54:11.904901 32350 raft_consensus.cc:686] T c76e757c44aa42008cf59d51027e0a3c P 80ec10c0466246e5a4fb75ce61877a1f [term 1 LEADER]: Becoming Leader. State: Replica: 80ec10c0466246e5a4fb75ce61877a1f, State: Running, Role: LEADER
I20260928 08:54:11.904951 32350 consensus_queue.cc:237] T c76e757c44aa42008cf59d51027e0a3c P 80ec10c0466246e5a4fb75ce61877a1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } }
I20260928 08:54:11.905644 31760 catalog_manager.cc:5495] T c76e757c44aa42008cf59d51027e0a3c P 80ec10c0466246e5a4fb75ce61877a1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80ec10c0466246e5a4fb75ce61877a1f (127.23.167.195). New cstate: current_term: 1 leader_uuid: "80ec10c0466246e5a4fb75ce61877a1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80ec10c0466246e5a4fb75ce61877a1f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 42447 } health_report { overall_health: HEALTHY } } }
I20260928 08:54:11.931726 32267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.929498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:51171 (local address 127.23.167.194:33563)
0928 08:54:11.929562 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:11.929563 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:11.929610 (+ 47us) server_negotiation.cc:407] Connection header received
0928 08:54:11.929674 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:11.929674 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:11.929686 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:11.929730 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:54:11.929849 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.930322 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.931075 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.931259 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.931313 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.931480 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:54:11.931511 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:54:11.931551 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:11.931607 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 08:54:11.931630 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 08:54:11.931734 32358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.929404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:33563 (local address 127.23.167.195:51171)
0928 08:54:11.929548 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:11.929560 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:11.929624 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:11.929731 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:11.929732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:11.929734 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:54:11.929789 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.929791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.930399 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.930399 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.930994 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.930996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.931337 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.931338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.931386 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.931400 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:54:11.931562 (+ 162us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:54:11.931564 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 08:54:11.931604 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 08:54:11.931656 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":79,"threads_started":1}
I20260928 08:54:11.933576 32057 raft_consensus.cc:1261] T c76e757c44aa42008cf59d51027e0a3c P b4a99b7bc9f54355980d72922c77af06 [term 1 FOLLOWER]: Refusing update from remote peer 80ec10c0466246e5a4fb75ce61877a1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:54:11.933604 31916 raft_consensus.cc:1261] T c76e757c44aa42008cf59d51027e0a3c P c7fa0d4b654942aca29b1488c9dacb1b [term 1 FOLLOWER]: Refusing update from remote peer 80ec10c0466246e5a4fb75ce61877a1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:54:11.933771 32350 consensus_queue.cc:1035] T c76e757c44aa42008cf59d51027e0a3c P 80ec10c0466246e5a4fb75ce61877a1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7fa0d4b654942aca29b1488c9dacb1b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:54:11.933876 32345 consensus_queue.cc:1035] T c76e757c44aa42008cf59d51027e0a3c P 80ec10c0466246e5a4fb75ce61877a1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4a99b7bc9f54355980d72922c77af06" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:54:11.940340 32265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.938219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:57029 (local address 127.23.167.195:42447)
0928 08:54:11.938258 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:11.938260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:11.938267 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:54:11.938321 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:11.938322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:11.938330 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:11.938377 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:54:11.938509 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.938974 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.939680 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.939897 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.939960 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.940091 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:54:11.940126 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:54:11.940172 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:11.940234 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 08:54:11.940257 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 08:54:11.940343 32328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.938181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:42447 (local address 127.23.167.194:57029)
0928 08:54:11.938221 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:11.938233 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:11.938276 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:11.938384 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:11.938386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:11.938387 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:54:11.938442 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.938443 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.939044 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.939044 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.939617 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:11.939618 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:11.939966 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:11.939967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:11.940021 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:11.940027 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:54:11.940185 (+ 158us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:54:11.940187 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 08:54:11.940226 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 08:54:11.940268 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 08:54:11.946439 31760 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:54:11.946534 31760 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:54:11.947094 31760 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc1b95ca7c2d48d5ae1c2c1064bcb2b1: Sending DeleteTablet for 3 replicas of tablet c76e757c44aa42008cf59d51027e0a3c
I20260928 08:54:11.947307 32037 tablet_service.cc:1514] Processing DeleteTablet for tablet c76e757c44aa42008cf59d51027e0a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:54:11 UTC) from {username='oryx', principal='oryx/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:56940
I20260928 08:54:11.947338 31897 tablet_service.cc:1514] Processing DeleteTablet for tablet c76e757c44aa42008cf59d51027e0a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:54:11 UTC) from {username='oryx', principal='oryx/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:37296
I20260928 08:54:11.947363 32177 tablet_service.cc:1514] Processing DeleteTablet for tablet c76e757c44aa42008cf59d51027e0a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:54:11 UTC) from {username='oryx', principal='oryx/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:46584
I20260928 08:54:11.947408 32292 tablet_replica.cc:330] T c76e757c44aa42008cf59d51027e0a3c P b4a99b7bc9f54355980d72922c77af06: stopping tablet replica
I20260928 08:54:11.947418 32294 tablet_replica.cc:330] T c76e757c44aa42008cf59d51027e0a3c P c7fa0d4b654942aca29b1488c9dacb1b: stopping tablet replica
I20260928 08:54:11.947461 32292 raft_consensus.cc:2229] T c76e757c44aa42008cf59d51027e0a3c P b4a99b7bc9f54355980d72922c77af06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:54:11.947492 32292 raft_consensus.cc:2258] T c76e757c44aa42008cf59d51027e0a3c P b4a99b7bc9f54355980d72922c77af06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:54:11.947487 32294 raft_consensus.cc:2229] T c76e757c44aa42008cf59d51027e0a3c P c7fa0d4b654942aca29b1488c9dacb1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:54:11.947518 32294 raft_consensus.cc:2258] T c76e757c44aa42008cf59d51027e0a3c P c7fa0d4b654942aca29b1488c9dacb1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:54:11.947589 32293 tablet_replica.cc:330] T c76e757c44aa42008cf59d51027e0a3c P 80ec10c0466246e5a4fb75ce61877a1f: stopping tablet replica
I20260928 08:54:11.947649 32293 raft_consensus.cc:2229] T c76e757c44aa42008cf59d51027e0a3c P 80ec10c0466246e5a4fb75ce61877a1f [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:54:11.947710 32293 raft_consensus.cc:2258] T c76e757c44aa42008cf59d51027e0a3c P 80ec10c0466246e5a4fb75ce61877a1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:54:11.947790 32294 ts_tablet_manager.cc:1905] T c76e757c44aa42008cf59d51027e0a3c P c7fa0d4b654942aca29b1488c9dacb1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:54:11.948004 32292 ts_tablet_manager.cc:1905] T c76e757c44aa42008cf59d51027e0a3c P b4a99b7bc9f54355980d72922c77af06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:54:11.948109 32293 ts_tablet_manager.cc:1905] T c76e757c44aa42008cf59d51027e0a3c P 80ec10c0466246e5a4fb75ce61877a1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:54:11.948202 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 31832
I20260928 08:54:11.949425 32292 ts_tablet_manager.cc:1918] T c76e757c44aa42008cf59d51027e0a3c P b4a99b7bc9f54355980d72922c77af06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:54:11.949460 32293 ts_tablet_manager.cc:1918] T c76e757c44aa42008cf59d51027e0a3c P 80ec10c0466246e5a4fb75ce61877a1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:54:11.949476 32292 log.cc:1196] T c76e757c44aa42008cf59d51027e0a3c P b4a99b7bc9f54355980d72922c77af06: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-1/wal/wals/c76e757c44aa42008cf59d51027e0a3c
I20260928 08:54:11.949506 32293 log.cc:1196] T c76e757c44aa42008cf59d51027e0a3c P 80ec10c0466246e5a4fb75ce61877a1f: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585589999956-24223-0/minicluster-data/ts-2/wal/wals/c76e757c44aa42008cf59d51027e0a3c
I20260928 08:54:11.949682 32292 ts_tablet_manager.cc:1939] T c76e757c44aa42008cf59d51027e0a3c P b4a99b7bc9f54355980d72922c77af06: Deleting consensus metadata
I20260928 08:54:11.949702 32293 ts_tablet_manager.cc:1939] T c76e757c44aa42008cf59d51027e0a3c P 80ec10c0466246e5a4fb75ce61877a1f: Deleting consensus metadata
I20260928 08:54:11.949985 31744 catalog_manager.cc:4841] TS 80ec10c0466246e5a4fb75ce61877a1f (127.23.167.195:42447): tablet c76e757c44aa42008cf59d51027e0a3c (table test-table [id=ba9159ca74dc496da425457990780dfd]) successfully deleted
I20260928 08:54:11.951118 31745 catalog_manager.cc:4841] TS b4a99b7bc9f54355980d72922c77af06 (127.23.167.194:33563): tablet c76e757c44aa42008cf59d51027e0a3c (table test-table [id=ba9159ca74dc496da425457990780dfd]) successfully deleted
W20260928 08:54:11.955168 31746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.167.193:46433 (error 108)
I20260928 08:54:11.955359 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 31972
I20260928 08:54:11.956097 32362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:11.955490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:46433 (local address 127.0.0.1:37314)
0928 08:54:11.956008 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:11.956054 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.193:46433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":36,"threads_started":1}
W20260928 08:54:11.956219 31746 catalog_manager.cc:4584] TS c7fa0d4b654942aca29b1488c9dacb1b (127.23.167.193:46433): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c76e757c44aa42008cf59d51027e0a3c: Network error: Client connection negotiation failed: client connection to 127.23.167.193:46433: connect: Connection refused (error 111)
I20260928 08:54:11.961169 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 32112
I20260928 08:54:11.965667 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 31728
2026-09-28T08:54:11Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2612 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:54:12 dist-test-slave-pbjp krb5kdc[32365](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:54:12 dist-test-slave-pbjp krb5kdc[32365](info): set up 2 sockets
Sep 28 08:54:12 dist-test-slave-pbjp krb5kdc[32365](info): commencing operation
krb5kdc: starting...
W20260928 08:54:12.925313 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:54:12 dist-test-slave-pbjp krb5kdc[32365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:54:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:54:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.167.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
I20260928 08:54:13.035750 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:38129
--webserver_interface=127.23.167.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:38129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:54:13.115917 32378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:54:13.116138 32378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:54:13.116170 32378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:54:13.117386 32378 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:54:13.117429 32378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:54:13.117455 32378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:54:13.117471 32378 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:54:13.117488 32378 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:54:13.119052 32378 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:38129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:38129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:54:13.119323 32378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:54:13.119536 32378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:54:13.124163 32383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:13.124152 32384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:13.124249 32386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:13.124467 32378 server_base.cc:1034] running on GCE node
I20260928 08:54:13.124691 32378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:54:13.124985 32378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:54:13.126157 32378 hybrid_clock.cc:648] HybridClock initialized: now 1790585653126151 us; error 42 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T08:54:13Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1167 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:54:13 dist-test-slave-pbjp krb5kdc[32391](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:54:13 dist-test-slave-pbjp krb5kdc[32391](info): set up 2 sockets
Sep 28 08:54:13 dist-test-slave-pbjp krb5kdc[32391](info): commencing operation
krb5kdc: starting...
W20260928 08:54:14.081164 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:54:14 dist-test-slave-pbjp krb5kdc[32391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:54:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:54:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.167.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
I20260928 08:54:14.196328 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:38859
--webserver_interface=127.23.167.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:38859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:54:14.277058 32404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:54:14.277300 32404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:54:14.277331 32404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:54:14.278510 32404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:54:14.278558 32404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:54:14.278571 32404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:54:14.278582 32404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:54:14.278594 32404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:54:14.280098 32404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:38859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:38859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:54:14.280346 32404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:54:14.280567 32404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:54:14.285208 32409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:14.285243 32404 server_base.cc:1034] running on GCE node
W20260928 08:54:14.285238 32410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:14.285223 32412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:14.285653 32404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:54:14.285936 32404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:54:14.287098 32404 hybrid_clock.cc:648] HybridClock initialized: now 1790585654287082 us; error 29 us; skew 500 ppm
Sep 28 08:54:14 dist-test-slave-pbjp krb5kdc[32391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585654, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:54:14.289507 32404 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:54:14.289948 32404 webserver.cc:460] Webserver started at http://127.23.167.254:42275/ using document root <none> and password file <none>
I20260928 08:54:14.290153 32404 fs_manager.cc:359] Metadata directory not provided
I20260928 08:54:14.290201 32404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:54:14.290323 32404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:54:14.291353 32404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "a216081023b345d28c118e1bc928c648"
format_stamp: "Formatted at 2026-09-28 08:54:14 on dist-test-slave-pbjp"
server_key: "3982a67210e6f6529e5a55feafc6f494"
server_key_iv: "0af078ad884c49047f1ce07810d28d9b"
server_key_version: "encryptionkey@0"
I20260928 08:54:14.291713 32404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "a216081023b345d28c118e1bc928c648"
format_stamp: "Formatted at 2026-09-28 08:54:14 on dist-test-slave-pbjp"
server_key: "3982a67210e6f6529e5a55feafc6f494"
server_key_iv: "0af078ad884c49047f1ce07810d28d9b"
server_key_version: "encryptionkey@0"
I20260928 08:54:14.292958 32404 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:54:14.293676 32419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:14.293825 32404 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:14.293900 32404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "a216081023b345d28c118e1bc928c648"
format_stamp: "Formatted at 2026-09-28 08:54:14 on dist-test-slave-pbjp"
server_key: "3982a67210e6f6529e5a55feafc6f494"
server_key_iv: "0af078ad884c49047f1ce07810d28d9b"
server_key_version: "encryptionkey@0"
I20260928 08:54:14.293983 32404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:54:14.311190 32404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:54:14.313009 32404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:54:14.313179 32404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:54:14.316937 32404 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:38859
I20260928 08:54:14.316980 32471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:38859 every 8 connection(s)
I20260928 08:54:14.317339 32404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:54:14.318199 32472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:54:14.320426 32472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648: Bootstrap starting.
I20260928 08:54:14.320988 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 32404
I20260928 08:54:14.321022 32472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648: Neither blocks nor log segments found. Creating new log.
I20260928 08:54:14.321111 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:54:14.321312 24223 external_mini_cluster.cc:1428] Setting key 13a88c583accdc78b4707fd485ecdebe
I20260928 08:54:14.321353 32472 log.cc:824] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648: Log is configured to *not* fsync() on all Append() calls
I20260928 08:54:14.322098 32472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648: No bootstrap required, opened a new log
I20260928 08:54:14.324579 32472 raft_consensus.cc:348] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a216081023b345d28c118e1bc928c648" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38859 } }
I20260928 08:54:14.324749 32472 raft_consensus.cc:374] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:54:14.324783 32472 raft_consensus.cc:729] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a216081023b345d28c118e1bc928c648, State: Initialized, Role: FOLLOWER
I20260928 08:54:14.324949 32472 consensus_queue.cc:260] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a216081023b345d28c118e1bc928c648" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38859 } }
I20260928 08:54:14.325033 32472 raft_consensus.cc:388] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:54:14.325090 32472 raft_consensus.cc:482] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:54:14.325143 32472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:54:14.325743 32472 raft_consensus.cc:504] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a216081023b345d28c118e1bc928c648" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38859 } }
I20260928 08:54:14.325850 32472 leader_election.cc:302] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a216081023b345d28c118e1bc928c648; no voters:
I20260928 08:54:14.326004 32472 leader_election.cc:288] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:54:14.326046 32477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:54:14.326236 32477 raft_consensus.cc:686] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [term 1 LEADER]: Becoming Leader. State: Replica: a216081023b345d28c118e1bc928c648, State: Running, Role: LEADER
I20260928 08:54:14.326256 32472 sys_catalog.cc:564] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:54:14.326423 32477 consensus_queue.cc:237] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a216081023b345d28c118e1bc928c648" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38859 } }
I20260928 08:54:14.326834 32478 sys_catalog.cc:455] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a216081023b345d28c118e1bc928c648" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a216081023b345d28c118e1bc928c648" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38859 } } }
I20260928 08:54:14.326910 32478 sys_catalog.cc:458] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [sys.catalog]: This master's current role is: LEADER
I20260928 08:54:14.327181 32479 sys_catalog.cc:455] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a216081023b345d28c118e1bc928c648. Latest consensus state: current_term: 1 leader_uuid: "a216081023b345d28c118e1bc928c648" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a216081023b345d28c118e1bc928c648" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38859 } } }
I20260928 08:54:14.327287 32479 sys_catalog.cc:458] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [sys.catalog]: This master's current role is: LEADER
I20260928 08:54:14.327314 32486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:54:14.327811 32486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 08:54:14 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:54:14.329809 32486 catalog_manager.cc:1347] Generated new cluster ID: d6343f6042704ea985325f01355baa5b
I20260928 08:54:14.329857 32486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:54:14.336467 32476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:14.321804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56604 (local address 127.23.167.254:38859)
0928 08:54:14.321997 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:14.322006 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:14.322040 (+ 34us) server_negotiation.cc:407] Connection header received
0928 08:54:14.322560 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:14.322569 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:14.322637 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:14.322790 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:14.323354 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:14.324738 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:14.325594 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:14.326218 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:14.326363 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:14.329343 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:54:14.329347 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:54:14.329347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:54:14.329359 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:54:14.334800 (+ 5441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:14.335224 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:14.335227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:14.335227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:14.335332 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:14.335536 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:14.335536 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:14.335537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:14.335854 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:54:14.335991 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:14.336108 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 08:54:14.336243 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":74,"threads_started":1}
I20260928 08:54:14.343144 32486 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:54:14.343884 32486 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:54:14.348925 32486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648: Generated new TSK 0
I20260928 08:54:14.349169 32486 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.167.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
I20260928 08:54:14.373772 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:0
--local_ip_for_outbound_sockets=127.23.167.193
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:38859
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:54:14.453455 32498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:54:14.453655 32498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:54:14.453677 32498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:54:14.454789 32498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:54:14.454823 32498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:54:14.454847 32498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:54:14.456195 32498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:38859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:54:14.456367 32498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:54:14.456538 32498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:54:14.461252 32506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:14.461323 32498 server_base.cc:1034] running on GCE node
W20260928 08:54:14.461244 32504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:14.461243 32503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:14.461689 32498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:54:14.461938 32498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:54:14.463088 32498 hybrid_clock.cc:648] HybridClock initialized: now 1790585654463072 us; error 28 us; skew 500 ppm
Sep 28 08:54:14 dist-test-slave-pbjp krb5kdc[32391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585654, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:54:14.468474 32498 init.cc:377] Logged in from keytab as kudu/127.23.167.193@KRBTEST.COM (short username kudu)
I20260928 08:54:14.468940 32498 webserver.cc:460] Webserver started at http://127.23.167.193:41749/ using document root <none> and password file <none>
I20260928 08:54:14.469190 32498 fs_manager.cc:359] Metadata directory not provided
I20260928 08:54:14.469244 32498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:54:14.469375 32498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:54:14.470430 32498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-0/data/instance:
uuid: "c5a2536d4ccc4922aed9198f5c50e679"
format_stamp: "Formatted at 2026-09-28 08:54:14 on dist-test-slave-pbjp"
server_key: "0031edd8850fe9174443be1a00db742e"
server_key_iv: "dc04b82d0af380fb01588dc9a650d462"
server_key_version: "encryptionkey@0"
I20260928 08:54:14.470758 32498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance:
uuid: "c5a2536d4ccc4922aed9198f5c50e679"
format_stamp: "Formatted at 2026-09-28 08:54:14 on dist-test-slave-pbjp"
server_key: "0031edd8850fe9174443be1a00db742e"
server_key_iv: "dc04b82d0af380fb01588dc9a650d462"
server_key_version: "encryptionkey@0"
I20260928 08:54:14.472007 32498 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 08:54:14.472703 32513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:14.472854 32498 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:54:14.472945 32498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "c5a2536d4ccc4922aed9198f5c50e679"
format_stamp: "Formatted at 2026-09-28 08:54:14 on dist-test-slave-pbjp"
server_key: "0031edd8850fe9174443be1a00db742e"
server_key_iv: "dc04b82d0af380fb01588dc9a650d462"
server_key_version: "encryptionkey@0"
I20260928 08:54:14.473021 32498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:54:14.486699 32498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:54:14.488478 32498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:54:14.488606 32498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:54:14.488811 32498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:54:14.489154 32498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:54:14.489190 32498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:14.489219 32498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:54:14.489243 32498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:14.494693 32498 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:42359
I20260928 08:54:14.494746 32626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:42359 every 8 connection(s)
I20260928 08:54:14.495030 32498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
I20260928 08:54:14.498116 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 32498
I20260928 08:54:14.498206 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance
I20260928 08:54:14.498364 24223 external_mini_cluster.cc:1428] Setting key 2a1bc7f2af25c33d6e6994302af15e04
Sep 28 08:54:14 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585654, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:54:14.504386 32476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:14.495673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:59983 (local address 127.23.167.254:38859)
0928 08:54:14.495731 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:14.495733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:14.495895 (+ 162us) server_negotiation.cc:407] Connection header received
0928 08:54:14.496378 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:14.496379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:14.496392 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:14.496458 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:14.497465 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:14.497973 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:14.499114 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:14.499332 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:14.499383 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:14.501519 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:54:14.501520 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:54:14.501521 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:54:14.501528 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:54:14.502825 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:14.503318 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:14.503319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:14.503319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:14.503368 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:14.503665 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:14.503667 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:14.503667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:14.503798 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:54:14.503863 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:14.504202 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 08:54:14.504280 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 08:54:14.504590 32629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:14.495637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.193:59983)
0928 08:54:14.495778 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:14.495819 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:14.496283 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:14.496557 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:14.496561 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:14.497166 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:14.497358 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:14.497363 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:14.498058 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:14.498059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:14.499042 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:14.499044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:14.499396 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:14.499397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:14.499521 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:14.499975 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:14.499983 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:14.501401 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:54:14.502934 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:54:14.502943 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:54:14.502945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:54:14.503242 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:54:14.503428 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:54:14.503429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:54:14.503429 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:54:14.503569 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:54:14.503868 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:54:14.503874 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:54:14.504056 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 08:54:14.504177 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 08:54:14.504348 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260928 08:54:14.505175 32627 heartbeater.cc:347] Connected to a master server at 127.23.167.254:38859
I20260928 08:54:14.505277 32627 heartbeater.cc:464] Registering TS with master...
I20260928 08:54:14.505452 32627 heartbeater.cc:511] Master 127.23.167.254:38859 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.167.194@KRBTEST.COM; defaulting to no policy
I20260928 08:54:14.506043 32436 ts_manager.cc:194] Registered new tserver with Master: c5a2536d4ccc4922aed9198f5c50e679 (127.23.167.193:42359)
I20260928 08:54:14.506716 32436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.193@KRBTEST.COM'} at 127.23.167.193:59983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
I20260928 08:54:14.524062 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:0
--local_ip_for_outbound_sockets=127.23.167.194
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:38859
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:54:14.603422 32632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:54:14.603647 32632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:54:14.603672 32632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:54:14.604884 32632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:54:14.604930 32632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:54:14.604959 32632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:54:14.606438 32632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:38859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:54:14.606673 32632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:54:14.606913 32632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:54:14.612164 32638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:14.612169 32640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:14.612167 32637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:14.612473 32632 server_base.cc:1034] running on GCE node
I20260928 08:54:14.612654 32632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:54:14.612895 32632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:54:14.614041 32632 hybrid_clock.cc:648] HybridClock initialized: now 1790585654614010 us; error 43 us; skew 500 ppm
Sep 28 08:54:14 dist-test-slave-pbjp krb5kdc[32391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585654, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:54:14.616524 32632 init.cc:377] Logged in from keytab as kudu/127.23.167.194@KRBTEST.COM (short username kudu)
I20260928 08:54:14.616966 32632 webserver.cc:460] Webserver started at http://127.23.167.194:35641/ using document root <none> and password file <none>
I20260928 08:54:14.617201 32632 fs_manager.cc:359] Metadata directory not provided
I20260928 08:54:14.617254 32632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:54:14.617383 32632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:54:14.618449 32632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-1/data/instance:
uuid: "b4c10f0066ec4d0aa171232da22f9682"
format_stamp: "Formatted at 2026-09-28 08:54:14 on dist-test-slave-pbjp"
server_key: "2df2d443487f6876d62f314f9b0e8353"
server_key_iv: "1975e91171ecfd1f9a7426b6af8d61a7"
server_key_version: "encryptionkey@0"
I20260928 08:54:14.618779 32632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance:
uuid: "b4c10f0066ec4d0aa171232da22f9682"
format_stamp: "Formatted at 2026-09-28 08:54:14 on dist-test-slave-pbjp"
server_key: "2df2d443487f6876d62f314f9b0e8353"
server_key_iv: "1975e91171ecfd1f9a7426b6af8d61a7"
server_key_version: "encryptionkey@0"
I20260928 08:54:14.620092 32632 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:54:14.620807 32647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:14.620965 32632 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:54:14.621057 32632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "b4c10f0066ec4d0aa171232da22f9682"
format_stamp: "Formatted at 2026-09-28 08:54:14 on dist-test-slave-pbjp"
server_key: "2df2d443487f6876d62f314f9b0e8353"
server_key_iv: "1975e91171ecfd1f9a7426b6af8d61a7"
server_key_version: "encryptionkey@0"
I20260928 08:54:14.621155 32632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:54:14.633198 32632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:54:14.635154 32632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:54:14.635296 32632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:54:14.635516 32632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:54:14.635818 32632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:54:14.635854 32632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:14.635882 32632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:54:14.635906 32632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:14.641513 32632 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:44249
I20260928 08:54:14.641593 32760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:44249 every 8 connection(s)
I20260928 08:54:14.641870 32632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
Sep 28 08:54:14 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585654, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:54:14.648620 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 32632
I20260928 08:54:14.648727 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance
I20260928 08:54:14.648905 24223 external_mini_cluster.cc:1428] Setting key 07d8fe696255425cfc051b65b124a979
I20260928 08:54:14.651577 32476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:14.642538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:39169 (local address 127.23.167.254:38859)
0928 08:54:14.642582 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:14.642584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:14.642811 (+ 227us) server_negotiation.cc:407] Connection header received
0928 08:54:14.643328 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:14.643330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:14.643343 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:14.643395 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:14.644403 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:14.645196 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:14.646180 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:14.646370 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:14.646434 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:14.648536 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:54:14.648538 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:54:14.648538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:54:14.648545 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:54:14.650097 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:14.650547 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:14.650548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:14.650548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:14.650595 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:14.650862 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:14.650864 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:14.650864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:14.651021 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:54:14.651100 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:14.651367 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 08:54:14.651483 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 08:54:14.651629 32763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:14.642519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.194:39169)
0928 08:54:14.642682 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:14.642720 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:14.643219 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:14.643468 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:14.643472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:14.644035 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:14.644293 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:14.644301 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:14.645271 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:14.645272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:14.646092 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:14.646094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:14.646446 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:14.646446 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:14.646559 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:14.646991 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:14.647000 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:14.648431 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:54:14.650177 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:54:14.650185 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:54:14.650188 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:54:14.650481 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:54:14.650648 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:54:14.650649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:54:14.650649 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:54:14.650779 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:54:14.651096 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:54:14.651102 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:54:14.651245 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 08:54:14.651331 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 08:54:14.651460 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
I20260928 08:54:14.652151 32761 heartbeater.cc:347] Connected to a master server at 127.23.167.254:38859
I20260928 08:54:14.652251 32761 heartbeater.cc:464] Registering TS with master...
I20260928 08:54:14.652433 32761 heartbeater.cc:511] Master 127.23.167.254:38859 requested a full tablet report, sending...
I20260928 08:54:14.652834 32436 ts_manager.cc:194] Registered new tserver with Master: b4c10f0066ec4d0aa171232da22f9682 (127.23.167.194:44249)
I20260928 08:54:14.653370 32436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.194@KRBTEST.COM'} at 127.23.167.194:39169
WARNING: no policy specified for kudu/127.23.167.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
I20260928 08:54:14.673224 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:0
--local_ip_for_outbound_sockets=127.23.167.195
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:38859
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:54:14.753818 32766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:54:14.754029 32766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:54:14.754051 32766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:54:14.755232 32766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:54:14.755276 32766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:54:14.755311 32766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:54:14.756745 32766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:38859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:54:14.756981 32766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:54:14.757222 32766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:54:14.762101 304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:14.762117 306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:14.762117 303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:14.762171 32766 server_base.cc:1034] running on GCE node
I20260928 08:54:14.762568 32766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:54:14.762810 32766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:54:14.763947 32766 hybrid_clock.cc:648] HybridClock initialized: now 1790585654763934 us; error 28 us; skew 500 ppm
Sep 28 08:54:14 dist-test-slave-pbjp krb5kdc[32391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585654, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:54:14.766579 32766 init.cc:377] Logged in from keytab as kudu/127.23.167.195@KRBTEST.COM (short username kudu)
I20260928 08:54:14.767045 32766 webserver.cc:460] Webserver started at http://127.23.167.195:35695/ using document root <none> and password file <none>
I20260928 08:54:14.767248 32766 fs_manager.cc:359] Metadata directory not provided
I20260928 08:54:14.767306 32766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:54:14.767431 32766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:54:14.768479 32766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-2/data/instance:
uuid: "56be219c038d431dbe5ea6339b99c809"
format_stamp: "Formatted at 2026-09-28 08:54:14 on dist-test-slave-pbjp"
server_key: "2f4ddca8da836e7922d40ace1f9e30d6"
server_key_iv: "d84a2863c4267184fe9624ca8ed44364"
server_key_version: "encryptionkey@0"
I20260928 08:54:14.768805 32766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance:
uuid: "56be219c038d431dbe5ea6339b99c809"
format_stamp: "Formatted at 2026-09-28 08:54:14 on dist-test-slave-pbjp"
server_key: "2f4ddca8da836e7922d40ace1f9e30d6"
server_key_iv: "d84a2863c4267184fe9624ca8ed44364"
server_key_version: "encryptionkey@0"
I20260928 08:54:14.770162 32766 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:54:14.771006 313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:14.771185 32766 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:54:14.771274 32766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "56be219c038d431dbe5ea6339b99c809"
format_stamp: "Formatted at 2026-09-28 08:54:14 on dist-test-slave-pbjp"
server_key: "2f4ddca8da836e7922d40ace1f9e30d6"
server_key_iv: "d84a2863c4267184fe9624ca8ed44364"
server_key_version: "encryptionkey@0"
I20260928 08:54:14.771376 32766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:54:14.786012 32766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:54:14.788064 32766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:54:14.788200 32766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:54:14.788415 32766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:54:14.788700 32766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:54:14.788733 32766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:14.788761 32766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:54:14.788784 32766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:14.794266 32766 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:41255
I20260928 08:54:14.794322 426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:41255 every 8 connection(s)
I20260928 08:54:14.794608 32766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
I20260928 08:54:14.797749 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 32766
I20260928 08:54:14.797864 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance
I20260928 08:54:14.798074 24223 external_mini_cluster.cc:1428] Setting key 0567f682f0a9445308fe20e435b41afc
Sep 28 08:54:14 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585654, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:54:14.803556 32476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:14.795228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:59203 (local address 127.23.167.254:38859)
0928 08:54:14.795289 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:14.795291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:14.795492 (+ 201us) server_negotiation.cc:407] Connection header received
0928 08:54:14.795995 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:14.795996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:14.796007 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:14.796072 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:14.797015 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:14.797561 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:14.798975 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:14.799166 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:14.799215 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:14.801145 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:54:14.801147 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:54:14.801147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:54:14.801155 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:54:14.802464 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:14.802798 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:14.802799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:14.802799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:14.802847 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:14.803021 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:14.803021 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:14.803021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:14.803115 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:54:14.803158 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:14.803406 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 08:54:14.803483 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 08:54:14.803772 429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:14.795196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.195:59203)
0928 08:54:14.795345 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:14.795385 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:14.795885 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:14.796158 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:14.796162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:14.796721 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:14.796911 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:14.796916 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:14.797654 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:14.797655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:14.798904 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:14.798907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:14.799221 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:14.799222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:14.799325 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:14.799748 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:14.799756 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:14.801028 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:54:14.802531 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:54:14.802536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:54:14.802538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:54:14.802743 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:54:14.802891 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:54:14.802891 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:54:14.802892 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:54:14.802979 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:54:14.803156 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:54:14.803158 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:54:14.803268 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 08:54:14.803358 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 08:54:14.803557 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260928 08:54:14.804191 427 heartbeater.cc:347] Connected to a master server at 127.23.167.254:38859
I20260928 08:54:14.804275 427 heartbeater.cc:464] Registering TS with master...
I20260928 08:54:14.804418 427 heartbeater.cc:511] Master 127.23.167.254:38859 requested a full tablet report, sending...
I20260928 08:54:14.804755 32436 ts_manager.cc:194] Registered new tserver with Master: 56be219c038d431dbe5ea6339b99c809 (127.23.167.195:41255)
I20260928 08:54:14.805176 32436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.195@KRBTEST.COM'} at 127.23.167.195:59203
I20260928 08:54:14.809708 24223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.167.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.167.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.167.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.193.keytab.
Entry for principal oryx/127.23.167.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.193.keytab.
WARNING: no policy specified for oryx/127.23.167.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.167.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.167.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.194.keytab.
Entry for principal oryx/127.23.167.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.194.keytab.
WARNING: no policy specified for oryx/127.23.167.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.167.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.167.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.195.keytab.
Entry for principal oryx/127.23.167.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.195.keytab.
I20260928 08:54:14.878028 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 32498
I20260928 08:54:14.882086 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 32632
I20260928 08:54:14.885895 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 32766
I20260928 08:54:14.889433 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 32404
I20260928 08:54:14.893726 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:38859
--webserver_interface=127.23.167.254
--webserver_port=42275
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:38859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:54:14.974779 436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:54:14.974982 436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:54:14.975003 436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:54:14.976168 436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:54:14.976212 436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:54:14.976225 436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:54:14.976236 436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:54:14.976248 436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:54:14.977731 436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:38859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:38859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=42275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:54:14.977958 436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:54:14.978155 436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:54:14.982637 444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:14.982676 442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:14.982725 436 server_base.cc:1034] running on GCE node
W20260928 08:54:14.982637 441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:14.983072 436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:54:14.983338 436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:54:14.984494 436 hybrid_clock.cc:648] HybridClock initialized: now 1790585654984476 us; error 34 us; skew 500 ppm
Sep 28 08:54:14 dist-test-slave-pbjp krb5kdc[32391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585654, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:54:14.986943 436 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:54:14.987363 436 webserver.cc:460] Webserver started at http://127.23.167.254:42275/ using document root <none> and password file <none>
I20260928 08:54:14.987578 436 fs_manager.cc:359] Metadata directory not provided
I20260928 08:54:14.987638 436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:54:14.988924 436 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:54:14.989669 451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:14.989832 436 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:14.989909 436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "a216081023b345d28c118e1bc928c648"
format_stamp: "Formatted at 2026-09-28 08:54:14 on dist-test-slave-pbjp"
server_key: "3982a67210e6f6529e5a55feafc6f494"
server_key_iv: "0af078ad884c49047f1ce07810d28d9b"
server_key_version: "encryptionkey@0"
I20260928 08:54:14.990249 436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:54:15.016470 436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:54:15.018679 436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:54:15.018824 436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:54:15.022641 436 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:38859
I20260928 08:54:15.022678 503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:38859 every 8 connection(s)
I20260928 08:54:15.023041 436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:54:15.024477 504 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:54:15.025092 504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648: Bootstrap starting.
I20260928 08:54:15.028316 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 436
I20260928 08:54:15.028671 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:42359
--local_ip_for_outbound_sockets=127.23.167.193
--tserver_master_addrs=127.23.167.254:38859
--webserver_port=41749
--webserver_interface=127.23.167.193
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.167.193
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:54:15.033618 504 log.cc:824] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648: Log is configured to *not* fsync() on all Append() calls
I20260928 08:54:15.035310 504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:54:15.035593 504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648: Bootstrap complete.
I20260928 08:54:15.037844 504 raft_consensus.cc:348] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a216081023b345d28c118e1bc928c648" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38859 } }
I20260928 08:54:15.038089 504 raft_consensus.cc:729] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a216081023b345d28c118e1bc928c648, State: Initialized, Role: FOLLOWER
I20260928 08:54:15.038244 504 consensus_queue.cc:260] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a216081023b345d28c118e1bc928c648" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38859 } }
I20260928 08:54:15.038314 504 raft_consensus.cc:388] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:54:15.038360 504 raft_consensus.cc:482] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:54:15.038407 504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [term 1 FOLLOWER]: Advancing to term 2
I20260928 08:54:15.039507 504 raft_consensus.cc:504] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a216081023b345d28c118e1bc928c648" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38859 } }
I20260928 08:54:15.039603 504 leader_election.cc:302] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a216081023b345d28c118e1bc928c648; no voters:
I20260928 08:54:15.039772 504 leader_election.cc:288] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 08:54:15.039827 508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 08:54:15.040061 504 sys_catalog.cc:564] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:54:15.040060 508 raft_consensus.cc:686] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [term 2 LEADER]: Becoming Leader. State: Replica: a216081023b345d28c118e1bc928c648, State: Running, Role: LEADER
I20260928 08:54:15.040261 508 consensus_queue.cc:237] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a216081023b345d28c118e1bc928c648" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38859 } }
I20260928 08:54:15.040571 510 sys_catalog.cc:455] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a216081023b345d28c118e1bc928c648. Latest consensus state: current_term: 2 leader_uuid: "a216081023b345d28c118e1bc928c648" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a216081023b345d28c118e1bc928c648" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38859 } } }
I20260928 08:54:15.040647 510 sys_catalog.cc:458] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [sys.catalog]: This master's current role is: LEADER
I20260928 08:54:15.040768 509 sys_catalog.cc:455] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a216081023b345d28c118e1bc928c648" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a216081023b345d28c118e1bc928c648" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38859 } } }
I20260928 08:54:15.040817 509 sys_catalog.cc:458] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648 [sys.catalog]: This master's current role is: LEADER
I20260928 08:54:15.041147 517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:54:15.041654 517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:54:15.041810 517 catalog_manager.cc:1259] Loaded cluster ID: d6343f6042704ea985325f01355baa5b
I20260928 08:54:15.041992 517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:54:15.043226 517 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:54:15.043434 517 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a216081023b345d28c118e1bc928c648: Loaded TSK: 0
I20260928 08:54:15.043682 517 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 08:54:15.111107 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:54:15.111305 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:54:15.111327 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:54:15.112501 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:54:15.112543 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:54:15.112568 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:54:15.114027 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.193.keytab
--principal=oryx/127.23.167.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:42359
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=41749
--tserver_master_addrs=127.23.167.254:38859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:54:15.114246 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:54:15.114471 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:54:15.119256 530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:15.119261 531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:15.119323 533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:15.119702 506 server_base.cc:1034] running on GCE node
I20260928 08:54:15.119887 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:54:15.120107 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:54:15.121243 506 hybrid_clock.cc:648] HybridClock initialized: now 1790585655121225 us; error 34 us; skew 500 ppm
Sep 28 08:54:15 dist-test-slave-pbjp krb5kdc[32391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585655, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:54:15.123864 506 init.cc:377] Logged in from keytab as oryx/127.23.167.193@KRBTEST.COM (short username oryx)
I20260928 08:54:15.124378 506 webserver.cc:460] Webserver started at http://127.23.167.193:41749/ using document root <none> and password file <none>
I20260928 08:54:15.124703 506 fs_manager.cc:359] Metadata directory not provided
I20260928 08:54:15.124774 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:54:15.126498 506 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:54:15.127199 540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:15.127367 506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:54:15.127465 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "c5a2536d4ccc4922aed9198f5c50e679"
format_stamp: "Formatted at 2026-09-28 08:54:14 on dist-test-slave-pbjp"
server_key: "0031edd8850fe9174443be1a00db742e"
server_key_iv: "dc04b82d0af380fb01588dc9a650d462"
server_key_version: "encryptionkey@0"
I20260928 08:54:15.127943 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:54:15.162961 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:54:15.165480 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:54:15.165627 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:54:15.165889 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:54:15.166226 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:54:15.166280 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:15.166313 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:54:15.166332 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:15.173032 506 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:42359
I20260928 08:54:15.173128 653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:42359 every 8 connection(s)
I20260928 08:54:15.173494 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
I20260928 08:54:15.174096 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 506
I20260928 08:54:15.174389 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:44249
--local_ip_for_outbound_sockets=127.23.167.194
--tserver_master_addrs=127.23.167.254:38859
--webserver_port=35641
--webserver_interface=127.23.167.194
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.167.194
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:54:15 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:15 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:15.182237 656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:15.174202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.193:45881)
0928 08:54:15.174529 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:15.174574 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:15.175075 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:15.175487 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:15.175492 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:15.176049 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:15.176241 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:15.176247 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.177753 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.177754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:15.178559 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:15.178561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.179126 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.179127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:15.179250 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:15.179775 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:15.179784 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:15.182087 (+ 2303us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":76,"threads_started":1}
W20260928 08:54:15.182446 654 heartbeater.cc:644] Failed to heartbeat to 127.23.167.254:38859 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.167.254:38859: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
W20260928 08:54:15.182475 657 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:15.174276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:45881 (local address 127.23.167.254:38859)
0928 08:54:15.174658 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:15.174667 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:15.174702 (+ 35us) server_negotiation.cc:407] Connection header received
0928 08:54:15.175261 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:15.175267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:15.175314 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:15.175430 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:15.176359 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.177666 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.178643 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.179049 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.179139 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:15.182277 (+ 3138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.193:45881: BlockingRecv error: recv got EOF from 127.23.167.193:45881 (error 108)
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":61,"threads_started":1}
Sep 28 08:54:15 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:15 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:15.187457 656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:15.182860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.193:35009)
0928 08:54:15.182986 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:15.182996 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:15.183053 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:15.183182 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:15.183182 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:15.183332 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:15.183390 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:15.183392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.184283 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.184284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:15.184782 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:15.184783 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.185269 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.185277 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:15.185341 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:15.185743 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:15.185748 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:15.187410 (+ 1662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105}
W20260928 08:54:15.187599 657 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:15.182931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:35009 (local address 127.23.167.254:38859)
0928 08:54:15.182983 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:15.182985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:15.183043 (+ 58us) server_negotiation.cc:407] Connection header received
0928 08:54:15.183112 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:15.183113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:15.183122 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:15.183159 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:15.183490 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.184198 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.184855 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.185183 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.185250 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:15.187508 (+ 2258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.193:35009: BlockingRecv error: recv got EOF from 127.23.167.193:35009 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 28 08:54:15 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:15 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:15.191833 656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:15.187709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.193:54319)
0928 08:54:15.187738 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:15.187748 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:15.187783 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:15.187931 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:15.187932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:15.188027 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:15.188053 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:15.188055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.188941 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.188942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:15.189453 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:15.189455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.189876 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.189876 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:15.189914 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:15.190267 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:15.190271 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:15.191787 (+ 1516us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":14}
W20260928 08:54:15.191936 657 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:15.187803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:54319 (local address 127.23.167.254:38859)
0928 08:54:15.187833 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:15.187835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:15.187842 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:54:15.187862 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:15.187863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:15.187872 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:15.187923 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:15.188109 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.188860 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.189517 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.189808 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.189867 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:15.191880 (+ 2013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.193:54319: BlockingRecv error: recv got EOF from 127.23.167.193:54319 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260928 08:54:15.192013 654 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 08:54:15.258229 658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:54:15.258447 658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:54:15.258471 658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:54:15.259621 658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:54:15.259660 658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:54:15.259685 658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:54:15.261142 658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.194.keytab
--principal=oryx/127.23.167.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:44249
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=35641
--tserver_master_addrs=127.23.167.254:38859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:54:15.261368 658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:54:15.261574 658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:54:15.266249 664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:15.266376 658 server_base.cc:1034] running on GCE node
W20260928 08:54:15.266433 663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:15.266312 666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:15.266719 658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:54:15.266927 658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:54:15.268082 658 hybrid_clock.cc:648] HybridClock initialized: now 1790585655268062 us; error 34 us; skew 500 ppm
Sep 28 08:54:15 dist-test-slave-pbjp krb5kdc[32391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585655, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:54:15.270484 658 init.cc:377] Logged in from keytab as oryx/127.23.167.194@KRBTEST.COM (short username oryx)
I20260928 08:54:15.270866 658 webserver.cc:460] Webserver started at http://127.23.167.194:35641/ using document root <none> and password file <none>
I20260928 08:54:15.271071 658 fs_manager.cc:359] Metadata directory not provided
I20260928 08:54:15.271126 658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:54:15.272374 658 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:54:15.272899 673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:15.273056 658 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:54:15.273159 658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "b4c10f0066ec4d0aa171232da22f9682"
format_stamp: "Formatted at 2026-09-28 08:54:14 on dist-test-slave-pbjp"
server_key: "2df2d443487f6876d62f314f9b0e8353"
server_key_iv: "1975e91171ecfd1f9a7426b6af8d61a7"
server_key_version: "encryptionkey@0"
I20260928 08:54:15.273512 658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:54:15.281136 658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:54:15.283164 658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:54:15.283281 658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:54:15.283488 658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:54:15.283757 658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:54:15.283787 658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:15.283808 658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:54:15.283833 658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:15.289050 658 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:44249
I20260928 08:54:15.289180 786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:44249 every 8 connection(s)
I20260928 08:54:15.289431 658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
I20260928 08:54:15.290172 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 658
I20260928 08:54:15.290520 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:41255
--local_ip_for_outbound_sockets=127.23.167.195
--tserver_master_addrs=127.23.167.254:38859
--webserver_port=35695
--webserver_interface=127.23.167.195
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.167.195
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:54:15 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:15 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:15.297278 789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:15.290070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.194:37603)
0928 08:54:15.290235 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:15.290285 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:15.290874 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:15.291131 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:15.291136 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:15.291814 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:15.292080 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:15.292086 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.293013 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.293014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:15.293928 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:15.293931 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.294405 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.294406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:15.294526 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:15.295006 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:15.295013 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:15.297131 (+ 2118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
W20260928 08:54:15.297298 657 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:15.290087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:37603 (local address 127.23.167.254:38859)
0928 08:54:15.290149 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:15.290151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:15.290365 (+ 214us) server_negotiation.cc:407] Connection header received
0928 08:54:15.290991 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:15.290992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:15.291005 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:15.291055 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:15.292163 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.292900 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.294026 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.294328 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.294402 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:15.297234 (+ 2832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.194:37603: BlockingRecv error: recv got EOF from 127.23.167.194:37603 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260928 08:54:15.297505 787 heartbeater.cc:644] Failed to heartbeat to 127.23.167.254:38859 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.167.254:38859: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Sep 28 08:54:15 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:15 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:15.301999 789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:15.297698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.194:34207)
0928 08:54:15.297742 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:15.297752 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:15.297789 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:15.297891 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:15.297891 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:15.298009 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:15.298053 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:15.298054 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.298642 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.298643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:15.299138 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:15.299139 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.299518 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.299519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:15.299604 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:15.300013 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:15.300034 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:15.301935 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 08:54:15.302475 657 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:15.297751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:34207 (local address 127.23.167.254:38859)
0928 08:54:15.297797 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:15.297799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:15.297807 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:54:15.297831 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:15.297832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:15.297840 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:15.297885 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:15.298114 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.298576 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.299203 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.299456 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.299512 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:15.302114 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.194:34207: BlockingRecv error: recv got EOF from 127.23.167.194:34207 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 08:54:15 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:15 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:15.307226 789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:15.302254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.194:46955)
0928 08:54:15.302282 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:15.302293 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:15.302355 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:15.302657 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:15.302658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:15.302834 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:15.302868 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:15.302869 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.303705 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.303706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:15.304445 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:15.304448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.304904 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.304904 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:15.304966 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:15.305403 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:15.305408 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:15.307163 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":13}
W20260928 08:54:15.307338 787 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 08:54:15.307492 657 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:15.302339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:46955 (local address 127.23.167.254:38859)
0928 08:54:15.302564 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:15.302565 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:15.302572 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:54:15.302592 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:15.302593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:15.302602 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:15.302652 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:15.302926 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.303618 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.304539 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.304791 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.304830 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:15.307427 (+ 2597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.194:46955: BlockingRecv error: recv got EOF from 127.23.167.194:46955 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":60,"threads_started":1}
W20260928 08:54:15.373677 790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:54:15.373878 790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:54:15.373906 790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:54:15.375092 790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:54:15.375136 790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:54:15.375170 790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:54:15.376609 790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/krb5kdc/oryx_127.23.167.195.keytab
--principal=oryx/127.23.167.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:41255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=35695
--tserver_master_addrs=127.23.167.254:38859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:54:15.376844 790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:54:15.377085 790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:54:15.381779 799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:15.381861 797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:15.381788 796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:15.381950 790 server_base.cc:1034] running on GCE node
I20260928 08:54:15.382177 790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:54:15.382387 790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:54:15.383555 790 hybrid_clock.cc:648] HybridClock initialized: now 1790585655383540 us; error 30 us; skew 500 ppm
Sep 28 08:54:15 dist-test-slave-pbjp krb5kdc[32391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585655, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.167.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:54:15.385895 790 init.cc:377] Logged in from keytab as oryx/127.23.167.195@KRBTEST.COM (short username oryx)
I20260928 08:54:15.386395 790 webserver.cc:460] Webserver started at http://127.23.167.195:35695/ using document root <none> and password file <none>
I20260928 08:54:15.386610 790 fs_manager.cc:359] Metadata directory not provided
I20260928 08:54:15.386668 790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:54:15.387936 790 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:54:15.388451 806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:15.388583 790 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:54:15.388662 790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "56be219c038d431dbe5ea6339b99c809"
format_stamp: "Formatted at 2026-09-28 08:54:14 on dist-test-slave-pbjp"
server_key: "2f4ddca8da836e7922d40ace1f9e30d6"
server_key_iv: "d84a2863c4267184fe9624ca8ed44364"
server_key_version: "encryptionkey@0"
I20260928 08:54:15.388971 790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:54:15.405678 790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:54:15.407447 790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:54:15.407593 790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:54:15.407809 790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:54:15.408105 790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:54:15.408138 790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:15.408167 790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:54:15.408183 790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:15.413551 790 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:41255
I20260928 08:54:15.413610 919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:41255 every 8 connection(s)
I20260928 08:54:15.413916 790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
I20260928 08:54:15.415635 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 790
Sep 28 08:54:15 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:15 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 08:54:15.421736 657 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:15.414565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:37363 (local address 127.23.167.254:38859)
0928 08:54:15.414651 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:15.414653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:15.414838 (+ 185us) server_negotiation.cc:407] Connection header received
0928 08:54:15.415291 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:15.415292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:15.415301 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:15.415347 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:15.416507 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.417230 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.418331 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.418596 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.418645 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:15.421672 (+ 3027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.195:37363: BlockingRecv error: recv got EOF from 127.23.167.195:37363 (error 108)
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":49}
I20260928 08:54:15.421787 922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:15.414566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.195:37363)
0928 08:54:15.414720 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:15.414755 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:15.415215 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:15.415406 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:15.415409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:15.416135 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:15.416421 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:15.416427 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.417301 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.417301 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:15.418257 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:15.418259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.418668 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.418669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:15.418794 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:15.419284 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:15.419303 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:15.421588 (+ 2285us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
W20260928 08:54:15.422025 920 heartbeater.cc:644] Failed to heartbeat to 127.23.167.254:38859 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.167.254:38859: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
I20260928 08:54:15.423200 791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:15.416097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56610 (local address 127.23.167.254:38859)
0928 08:54:15.416148 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:15.416151 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:15.416187 (+ 36us) server_negotiation.cc:407] Connection header received
0928 08:54:15.416256 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:15.416257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:15.416269 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:15.416333 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:15.416994 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.417664 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.418406 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.418608 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.418665 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:15.419757 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:54:15.419762 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:54:15.419762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:54:15.419774 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:54:15.421642 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:15.422041 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:15.422044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:15.422044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:15.422338 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:15.422548 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:15.422549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:15.422549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:15.422802 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:54:15.422940 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:15.423011 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 08:54:15.423113 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":28}
Sep 28 08:54:15 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:15 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:15.426205 922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:15.422218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.195:59501)
0928 08:54:15.422269 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:15.422279 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:15.422327 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:15.422423 (+ 96us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:15.422423 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:15.422585 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:15.422630 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:15.422631 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.423416 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.423417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:15.424043 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:15.424045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.424337 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.424338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:15.424381 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:15.424746 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:15.424751 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:15.426163 (+ 1412us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 08:54:15.426287 657 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:15.422255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:59501 (local address 127.23.167.254:38859)
0928 08:54:15.422291 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:15.422293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:15.422314 (+ 21us) server_negotiation.cc:407] Connection header received
0928 08:54:15.422364 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:15.422365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:15.422374 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:15.422421 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:15.422687 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.423346 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.424116 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.424289 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.424333 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:15.426234 (+ 1901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.195:59501: BlockingRecv error: recv got EOF from 127.23.167.195:59501 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 28 08:54:15 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:15 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:15.429896 922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:15.426440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.195:53395)
0928 08:54:15.426485 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:15.426495 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:15.426535 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:15.426637 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:15.426637 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:15.426743 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:15.426765 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:15.426766 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.427325 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.427326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:15.427794 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:15.427795 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.428103 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.428104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:15.428150 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:15.428434 (+ 284us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:15.428437 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:15.429818 (+ 1381us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 08:54:15.429943 657 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:15.426490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:53395 (local address 127.23.167.254:38859)
0928 08:54:15.426524 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:15.426526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:15.426533 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:54:15.426588 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:15.426589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:15.426596 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:15.426635 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:15.426822 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.427262 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.427856 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:15.428049 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:15.428098 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:15.429897 (+ 1799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.195:53395: BlockingRecv error: recv got EOF from 127.23.167.195:53395 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 08:54:15.430042 920 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 08:54:16 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:16 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:16.197232 924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:16.192386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.193:40699)
0928 08:54:16.192511 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:16.192523 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:16.192578 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:16.192759 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:16.192760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:16.192963 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:16.193015 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:16.193017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:16.193751 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:16.193752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:16.194312 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:16.194313 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:16.194638 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:16.194638 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:16.194679 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:16.195189 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:16.195194 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:16.197167 (+ 1973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":66,"threads_started":1}
W20260928 08:54:16.197330 925 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:16.192488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:40699 (local address 127.23.167.254:38859)
0928 08:54:16.192610 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:16.192612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:16.192620 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:54:16.192641 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:16.192641 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:16.192651 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:16.192712 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:16.193116 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:16.193679 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:16.194375 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:16.194579 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:16.194630 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:16.197258 (+ 2628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.193:40699: BlockingRecv error: recv got EOF from 127.23.167.193:40699 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":68,"threads_started":1}
Sep 28 08:54:16 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:16 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:16.312071 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:16.307745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.194:55607)
0928 08:54:16.307863 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:16.307874 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:16.307942 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:16.308071 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:16.308072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:16.308219 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:16.308268 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:16.308270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:16.308888 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:16.308889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:16.309454 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:16.309455 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:16.309827 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:16.309828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:16.309910 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:16.310365 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:16.310370 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:16.311988 (+ 1618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":56,"threads_started":1}
W20260928 08:54:16.312134 925 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:16.307814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:55607 (local address 127.23.167.254:38859)
0928 08:54:16.307890 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:16.307892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:16.307923 (+ 31us) server_negotiation.cc:407] Connection header received
0928 08:54:16.307996 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:16.307997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:16.308007 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:16.308056 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:16.308327 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:16.308825 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:16.309526 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:16.309756 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:16.309815 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:16.312083 (+ 2268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.194:55607: BlockingRecv error: recv got EOF from 127.23.167.194:55607 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 08:54:16 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:16 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:16.435122 927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:16.430428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.195:33827)
0928 08:54:16.430554 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:16.430567 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:16.430645 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:16.430772 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:16.430773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:16.430942 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:16.430997 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:16.431003 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:16.431671 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:16.431672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:16.432277 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:16.432280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:16.432625 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:16.432626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:16.432690 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:16.433218 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:16.433222 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:16.435055 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
W20260928 08:54:16.435276 925 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:16.430520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:33827 (local address 127.23.167.254:38859)
0928 08:54:16.430592 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:16.430594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:16.430631 (+ 37us) server_negotiation.cc:407] Connection header received
0928 08:54:16.430704 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:16.430705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:16.430718 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:16.430766 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:16.431073 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:16.431589 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:16.432357 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:16.432564 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:16.432619 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:16.435204 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.195:33827: BlockingRecv error: recv got EOF from 127.23.167.195:33827 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 08:54:17 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:17 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:17.202207 928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:17.197680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.193:37093)
0928 08:54:17.197800 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:17.197813 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:17.197904 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:17.198037 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:17.198037 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:17.198192 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:17.198241 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:17.198243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:17.198940 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:17.198941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:17.199495 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:17.199497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:17.199860 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:17.199861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:17.199923 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:17.200365 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:17.200370 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:17.202148 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
W20260928 08:54:17.202310 929 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:17.197776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:37093 (local address 127.23.167.254:38859)
0928 08:54:17.197916 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:17.197918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:17.197927 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:54:17.197951 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:17.197952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:17.197962 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:17.198025 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:17.198339 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:17.198844 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:17.199591 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:17.199790 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:17.199856 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:17.202241 (+ 2385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.193:37093: BlockingRecv error: recv got EOF from 127.23.167.193:37093 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":65,"threads_started":1}
Sep 28 08:54:17 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:17 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:17.316907 930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:17.312597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.194:52697)
0928 08:54:17.312755 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:17.312765 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:17.312826 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:17.312973 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:17.312973 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:17.313165 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:17.313219 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:17.313221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:17.313874 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:17.313875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:17.314395 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:17.314397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:17.314727 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:17.314727 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:17.314783 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:17.315165 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:17.315169 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:17.316847 (+ 1678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":88,"threads_started":1}
W20260928 08:54:17.317003 929 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:17.312674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:52697 (local address 127.23.167.254:38859)
0928 08:54:17.312726 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:17.312728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:17.312820 (+ 92us) server_negotiation.cc:407] Connection header received
0928 08:54:17.312873 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:17.312874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:17.312885 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:17.312944 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:17.313296 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:17.313803 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:17.314468 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:17.314669 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:17.314721 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:17.316943 (+ 2222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.194:52697: BlockingRecv error: recv got EOF from 127.23.167.194:52697 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 08:54:17 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:17 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:17.440213 931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:17.435589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.195:37275)
0928 08:54:17.435756 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:17.435768 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:17.435840 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:17.435960 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:17.435961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:17.436115 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:17.436165 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:17.436167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:17.436758 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:17.436759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:17.437342 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:17.437346 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:17.437712 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:17.437713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:17.437765 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:17.438263 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:17.438267 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:17.440106 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":75,"threads_started":1}
W20260928 08:54:17.440358 929 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:17.435732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:37275 (local address 127.23.167.254:38859)
0928 08:54:17.435786 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:17.435788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:17.435816 (+ 28us) server_negotiation.cc:407] Connection header received
0928 08:54:17.435884 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:17.435885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:17.435898 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:17.435945 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:17.436216 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:17.436704 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:17.437425 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:17.437622 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:17.437680 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:17.440281 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.195:37275: BlockingRecv error: recv got EOF from 127.23.167.195:37275 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 28 08:54:18 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:18 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:18.207325 932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:18.202723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.193:36305)
0928 08:54:18.202865 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:18.202875 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:18.202934 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:18.203138 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:18.203139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:18.203300 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:18.203352 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:18.203353 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:18.204095 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:18.204096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:18.204655 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:18.204656 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:18.204978 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:18.204979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:18.205033 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:18.205518 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:18.205523 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:18.207238 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
W20260928 08:54:18.207406 933 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:18.202852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:36305 (local address 127.23.167.254:38859)
0928 08:54:18.202959 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:18.202961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:18.202969 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:54:18.202992 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:18.202992 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:18.203001 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:18.203062 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:18.203491 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:18.204009 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:18.204722 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:18.204912 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:18.204963 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:18.207343 (+ 2380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.193:36305: BlockingRecv error: recv got EOF from 127.23.167.193:36305 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":44,"threads_started":1}
Sep 28 08:54:18 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:18 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:18.321750 934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:18.317352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.194:52395)
0928 08:54:18.317484 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:18.317497 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:18.317573 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:18.317711 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:18.317712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:18.317878 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:18.317926 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:18.317928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:18.318589 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:18.318590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:18.319172 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:18.319175 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:18.319516 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:18.319517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:18.319572 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:18.319997 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:18.320002 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:18.321680 (+ 1678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":65,"threads_started":1}
W20260928 08:54:18.321837 933 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:18.317430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:52395 (local address 127.23.167.254:38859)
0928 08:54:18.317497 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:18.317500 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:18.317561 (+ 61us) server_negotiation.cc:407] Connection header received
0928 08:54:18.317632 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:18.317633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:18.317646 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:18.317695 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:18.317999 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:18.318509 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:18.319248 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:18.319456 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:18.319511 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:18.321776 (+ 2265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.194:52395: BlockingRecv error: recv got EOF from 127.23.167.194:52395 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 08:54:18 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:18 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:18.444994 935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:18.440665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.195:45241)
0928 08:54:18.440818 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:18.440828 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:18.440881 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:18.441011 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:18.441012 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:18.441189 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:18.441239 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:18.441241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:18.441913 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:18.441914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:18.442475 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:18.442477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:18.442771 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:18.442771 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:18.442808 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:18.443210 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:18.443214 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:18.444907 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":71,"threads_started":1}
W20260928 08:54:18.445055 933 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:18.440719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:45241 (local address 127.23.167.254:38859)
0928 08:54:18.440773 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:18.440775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:18.440879 (+ 104us) server_negotiation.cc:407] Connection header received
0928 08:54:18.440926 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:18.440927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:18.440939 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:18.440991 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:18.441300 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:18.441849 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:18.442538 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:18.442722 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:18.442768 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:18.445005 (+ 2237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.195:45241: BlockingRecv error: recv got EOF from 127.23.167.195:45241 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 28 08:54:19 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:19 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:19.212423 936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:19.207974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.193:48917)
0928 08:54:19.208134 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:19.208145 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:19.208197 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:19.208335 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:19.208336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:19.208497 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:19.208548 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:19.208550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:19.209212 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:19.209213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:19.209764 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:19.209765 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:19.210179 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:19.210181 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:19.210242 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:19.210685 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:19.210690 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:19.212361 (+ 1671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":83,"threads_started":1}
W20260928 08:54:19.212636 937 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:19.208090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:48917 (local address 127.23.167.254:38859)
0928 08:54:19.208205 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:19.208207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:19.208215 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:54:19.208239 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:19.208240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:19.208250 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:19.208314 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:19.208606 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:19.209151 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:19.209838 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:19.210066 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:19.210117 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:19.212559 (+ 2442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.193:48917: BlockingRecv error: recv got EOF from 127.23.167.193:48917 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":53,"threads_started":1}
Sep 28 08:54:19 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:19 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:19.326627 938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:19.322244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.194:38733)
0928 08:54:19.322389 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:19.322399 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:19.322469 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:19.322608 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:19.322608 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:19.322767 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:19.322817 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:19.322818 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:19.323405 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:19.323405 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:19.324043 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:19.324046 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:19.324416 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:19.324418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:19.324479 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:19.324901 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:19.324907 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:19.326549 (+ 1642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":87,"threads_started":1}
W20260928 08:54:19.326707 937 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:19.322324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:38733 (local address 127.23.167.254:38859)
0928 08:54:19.322380 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:19.322382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:19.322459 (+ 77us) server_negotiation.cc:407] Connection header received
0928 08:54:19.322522 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:19.322523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:19.322535 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:19.322596 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:19.322867 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:19.323353 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:19.324123 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:19.324346 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:19.324398 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:19.326655 (+ 2257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.194:38733: BlockingRecv error: recv got EOF from 127.23.167.194:38733 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 28 08:54:19 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:19 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:19.449740 939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:19.445498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.195:49815)
0928 08:54:19.445637 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:19.445647 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:19.445704 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:19.445826 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:19.445827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:19.445979 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:19.446026 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:19.446027 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:19.446648 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:19.446649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:19.447183 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:19.447184 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:19.447505 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:19.447506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:19.447558 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:19.448032 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:19.448036 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:19.449680 (+ 1644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
W20260928 08:54:19.449821 937 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:19.445572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:49815 (local address 127.23.167.254:38859)
0928 08:54:19.445630 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:19.445632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:19.445697 (+ 65us) server_negotiation.cc:407] Connection header received
0928 08:54:19.445757 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:19.445758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:19.445771 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:19.445808 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:19.446082 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:19.446576 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:19.447240 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:19.447412 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:19.447471 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:19.449757 (+ 2286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.195:49815: BlockingRecv error: recv got EOF from 127.23.167.195:49815 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 28 08:54:20 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:20 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:20.217828 940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:20.213016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.193:34483)
0928 08:54:20.213183 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:20.213194 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:20.213257 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:20.213403 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:20.213403 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:20.213568 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:20.213617 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:20.213619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:20.214288 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:20.214290 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:20.215043 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:20.215046 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:20.215422 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:20.215423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:20.215477 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:20.215925 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:20.215931 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:20.217769 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":114,"threads_started":1}
W20260928 08:54:20.217957 941 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:20.213158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:34483 (local address 127.23.167.254:38859)
0928 08:54:20.213272 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:20.213275 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:20.213285 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:54:20.213313 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:20.213314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:20.213324 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:20.213394 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:20.213675 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:20.214198 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:20.215123 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:20.215350 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:20.215414 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:20.217877 (+ 2463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.193:34483: BlockingRecv error: recv got EOF from 127.23.167.193:34483 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":48,"threads_started":1}
Sep 28 08:54:20 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:20 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:20.331530 942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:20.327117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.194:49077)
0928 08:54:20.327286 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:20.327303 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:20.327385 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:20.327521 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:20.327521 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:20.327689 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:20.327736 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:20.327737 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:20.328426 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:20.328427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:20.328958 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:20.328960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:20.329358 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:20.329359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:20.329416 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:20.329838 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:20.329843 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:20.331473 (+ 1630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
W20260928 08:54:20.331645 941 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:20.327246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:49077 (local address 127.23.167.254:38859)
0928 08:54:20.327302 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:20.327304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:20.327371 (+ 67us) server_negotiation.cc:407] Connection header received
0928 08:54:20.327448 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:20.327449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:20.327462 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:20.327510 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:20.327809 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:20.328311 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:20.329037 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:20.329278 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:20.329332 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:20.331582 (+ 2250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.194:49077: BlockingRecv error: recv got EOF from 127.23.167.194:49077 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 28 08:54:20 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:20 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:20.454718 943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:20.450258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.195:44415)
0928 08:54:20.450417 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:20.450433 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:20.450513 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:20.450649 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:20.450650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:20.450818 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:20.450866 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:20.450868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:20.451528 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:20.451529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:20.452143 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:20.452146 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:20.452546 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:20.452547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:20.452592 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:20.453039 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:20.453043 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:20.454661 (+ 1618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":67,"threads_started":1}
W20260928 08:54:20.454818 941 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:20.450382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:44415 (local address 127.23.167.254:38859)
0928 08:54:20.450472 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:20.450474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:20.450488 (+ 14us) server_negotiation.cc:407] Connection header received
0928 08:54:20.450579 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:20.450580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:20.450595 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:20.450644 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:20.450940 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:20.451442 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:20.452213 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:20.452473 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:20.452523 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:20.454752 (+ 2229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.195:44415: BlockingRecv error: recv got EOF from 127.23.167.195:44415 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 08:54:21 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:21 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:21.222535 944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:21.218254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.193:56303)
0928 08:54:21.218380 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:21.218390 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:21.218438 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:21.218567 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:21.218568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:21.218719 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:21.218783 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:21.218785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:21.219503 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:21.219504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:21.220038 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:21.220040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:21.220330 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:21.220330 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:21.220380 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:21.220766 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:21.220770 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:21.222478 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":58,"threads_started":1}
W20260928 08:54:21.222685 945 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:21.218299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:56303 (local address 127.23.167.254:38859)
0928 08:54:21.218420 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:21.218422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:21.218430 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:54:21.218494 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:21.218494 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:21.218504 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:21.218554 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:21.218833 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:21.219425 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:21.220096 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:21.220283 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:21.220326 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:21.222613 (+ 2287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.193:56303: BlockingRecv error: recv got EOF from 127.23.167.193:56303 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":67,"threads_started":1}
Sep 28 08:54:21 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:21 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:21.336477 946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:21.332028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.194:53771)
0928 08:54:21.332162 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:21.332173 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:21.332249 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:21.332382 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:21.332383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:21.332564 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:21.332614 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:21.332616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:21.333364 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:21.333365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:21.334056 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:21.334060 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:21.334368 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:21.334369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:21.334418 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:21.334822 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:21.334826 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:21.336400 (+ 1574us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
W20260928 08:54:21.336573 945 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:21.332097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:53771 (local address 127.23.167.254:38859)
0928 08:54:21.332157 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:21.332160 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:21.332228 (+ 68us) server_negotiation.cc:407] Connection header received
0928 08:54:21.332316 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:21.332317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:21.332330 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:21.332372 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:21.332701 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:21.333280 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:21.334116 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:21.334311 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:21.334364 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:21.336518 (+ 2154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.194:53771: BlockingRecv error: recv got EOF from 127.23.167.194:53771 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 28 08:54:21 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:21 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:21.459471 947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:21.455199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.195:34405)
0928 08:54:21.455316 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:21.455327 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:21.455385 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:21.455502 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:21.455503 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:21.455649 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:21.455710 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:21.455712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:21.456316 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:21.456316 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:21.456878 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:21.456880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:21.457204 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:21.457205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:21.457254 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:21.457704 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:21.457709 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:21.459405 (+ 1696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":61,"threads_started":1}
W20260928 08:54:21.459573 945 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:21.455294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:34405 (local address 127.23.167.254:38859)
0928 08:54:21.455366 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:21.455368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:21.455379 (+ 11us) server_negotiation.cc:407] Connection header received
0928 08:54:21.455434 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:21.455434 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:21.455447 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:21.455490 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:21.455760 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:21.456261 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:21.456947 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:21.457154 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:21.457204 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:21.459503 (+ 2299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.195:34405: BlockingRecv error: recv got EOF from 127.23.167.195:34405 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 08:54:22 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:22 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:22.227381 948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:22.223041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.193:56089)
0928 08:54:22.223191 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:22.223204 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:22.223267 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:22.223423 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:22.223424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:22.223580 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:22.223628 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:22.223630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:22.224266 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:22.224267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:22.224807 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:22.224808 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:22.225166 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:22.225167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:22.225208 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:22.225597 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:22.225601 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:22.227313 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":71,"threads_started":1}
W20260928 08:54:22.227480 949 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:22.223135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:56089 (local address 127.23.167.254:38859)
0928 08:54:22.223261 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:22.223263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:22.223271 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:54:22.223310 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:22.223311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:22.223321 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:22.223378 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:22.223685 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:22.224211 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:22.224875 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:22.225109 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:22.225159 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:22.227409 (+ 2250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.193:56089: BlockingRecv error: recv got EOF from 127.23.167.193:56089 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":66,"threads_started":1}
Sep 28 08:54:22 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:22 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:22.341760 950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:22.336997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.194:35233)
0928 08:54:22.337193 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:22.337207 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:22.337278 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:22.337417 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:22.337418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:22.337573 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:22.337624 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:22.337626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:22.338250 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:22.338251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:22.338807 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:22.338809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:22.339175 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:22.339176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:22.339225 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:22.339748 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:22.339752 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:22.341682 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":115,"threads_started":1}
W20260928 08:54:22.341845 949 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:22.337149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:35233 (local address 127.23.167.254:38859)
0928 08:54:22.337200 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:22.337201 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:22.337272 (+ 71us) server_negotiation.cc:407] Connection header received
0928 08:54:22.337345 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:22.337346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:22.337358 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:22.337397 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:22.337679 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:22.338181 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:22.338901 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:22.339096 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:22.339158 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:22.341791 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.194:35233: BlockingRecv error: recv got EOF from 127.23.167.194:35233 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 28 08:54:22 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:22 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:22.464187 951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:22.459942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.195:47991)
0928 08:54:22.460088 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:22.460098 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:22.460151 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:22.460272 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:22.460272 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:22.460432 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:22.460482 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:22.460484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:22.461139 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:22.461141 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:22.461676 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:22.461677 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:22.462004 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:22.462005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:22.462057 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:22.462475 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:22.462480 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:22.464127 (+ 1647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
W20260928 08:54:22.464365 949 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:22.460057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:47991 (local address 127.23.167.254:38859)
0928 08:54:22.460121 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:22.460123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:22.460134 (+ 11us) server_negotiation.cc:407] Connection header received
0928 08:54:22.460202 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:22.460203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:22.460214 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:22.460261 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:22.460543 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:22.461039 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:22.461738 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:22.461923 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:22.461971 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:22.464249 (+ 2278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.195:47991: BlockingRecv error: recv got EOF from 127.23.167.195:47991 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 08:54:23 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:23 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:23.232108 952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:23.227806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.193:36995)
0928 08:54:23.227948 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:23.227959 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:23.228016 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:23.228140 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:23.228141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:23.228297 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:23.228346 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:23.228348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:23.229000 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:23.229001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:23.229580 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:23.229583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:23.229937 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:23.229938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:23.229974 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:23.230379 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:23.230383 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:23.232049 (+ 1666us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
W20260928 08:54:23.232245 953 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:23.227900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:36995 (local address 127.23.167.254:38859)
0928 08:54:23.228000 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:23.228002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:23.228011 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:54:23.228062 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:23.228062 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:23.228072 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:23.228128 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:23.228404 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:23.228928 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:23.229656 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:23.229855 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:23.229934 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:23.232172 (+ 2238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.193:36995: BlockingRecv error: recv got EOF from 127.23.167.193:36995 (error 108)
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":49,"threads_started":1}
Sep 28 08:54:23 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:23 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:23.347052 954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:23.342241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.194:51193)
0928 08:54:23.342414 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:23.342424 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:23.342489 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:23.342622 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:23.342622 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:23.342779 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:23.342833 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:23.342835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:23.343600 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:23.343601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:23.344134 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:23.344136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:23.344548 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:23.344549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:23.344606 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:23.345037 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:23.345041 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:23.346993 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":119,"threads_started":1}
W20260928 08:54:23.347147 953 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:23.342306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:51193 (local address 127.23.167.254:38859)
0928 08:54:23.342420 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:23.342423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:23.342485 (+ 62us) server_negotiation.cc:407] Connection header received
0928 08:54:23.342552 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:23.342553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:23.342565 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:23.342609 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:23.342883 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:23.343519 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:23.344212 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:23.344457 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:23.344512 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:23.347088 (+ 2576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.194:51193: BlockingRecv error: recv got EOF from 127.23.167.194:51193 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 08:54:23 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:23 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:23.468924 955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:23.464660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.195:33863)
0928 08:54:23.464803 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:23.464814 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:23.464885 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:23.465049 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:23.465050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:23.465233 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:23.465280 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:23.465282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:23.465934 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:23.465935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:23.466464 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:23.466466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:23.466785 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:23.466786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:23.466838 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:23.467278 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:23.467282 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:23.468867 (+ 1585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":66,"threads_started":1}
W20260928 08:54:23.469100 953 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:23.464685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:33863 (local address 127.23.167.254:38859)
0928 08:54:23.464745 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:23.464747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:23.464923 (+ 176us) server_negotiation.cc:407] Connection header received
0928 08:54:23.464947 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:23.464948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:23.464963 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:23.465009 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:23.465356 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:23.465855 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:23.466543 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:23.466726 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:23.466779 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:23.468977 (+ 2198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.195:33863: BlockingRecv error: recv got EOF from 127.23.167.195:33863 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 28 08:54:24 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:24 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:24.237339 956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:24.232546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.193:37913)
0928 08:54:24.232691 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:24.232702 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:24.232757 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:24.232906 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:24.232906 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:24.233098 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:24.233152 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:24.233155 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:24.233921 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:24.233922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:24.234553 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:24.234557 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:24.234914 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:24.234915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:24.234974 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:24.235425 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:24.235429 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:24.237279 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":67,"threads_started":1}
W20260928 08:54:24.237494 957 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:24.232633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:37913 (local address 127.23.167.254:38859)
0928 08:54:24.232772 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:24.232775 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:24.232785 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:54:24.232812 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:24.232813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:24.232823 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:24.232894 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:24.233249 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:24.233811 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:24.234634 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:24.234835 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:24.234891 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:24.237418 (+ 2527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.193:37913: BlockingRecv error: recv got EOF from 127.23.167.193:37913 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
Sep 28 08:54:24 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:24 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:24.351986 958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:24.347524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.194:55935)
0928 08:54:24.347671 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:24.347682 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:24.347742 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:24.347891 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:24.347892 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:24.348048 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:24.348098 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:24.348100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:24.348720 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:24.348721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:24.349331 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:24.349335 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:24.349658 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:24.349658 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:24.349696 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:24.350135 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:24.350139 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:24.351899 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":73,"threads_started":1}
W20260928 08:54:24.352072 957 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:24.347629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:55935 (local address 127.23.167.254:38859)
0928 08:54:24.347687 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:24.347689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:24.347739 (+ 50us) server_negotiation.cc:407] Connection header received
0928 08:54:24.347815 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:24.347816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:24.347830 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:24.347875 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:24.348150 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:24.348649 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:24.349405 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:24.349602 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:24.349657 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:24.352010 (+ 2353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.194:55935: BlockingRecv error: recv got EOF from 127.23.167.194:55935 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 28 08:54:24 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:24 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:24.473927 959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:24.469375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.195:59467)
0928 08:54:24.469486 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:24.469498 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:24.469559 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:24.469686 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:24.469686 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:24.469840 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:24.469892 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:24.469894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:24.470493 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:24.470494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:24.471015 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:24.471016 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:24.471405 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:24.471405 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:24.471459 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:24.471892 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:24.471896 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:24.473849 (+ 1953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":56,"threads_started":1}
W20260928 08:54:24.474009 957 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:24.469460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:59467 (local address 127.23.167.254:38859)
0928 08:54:24.469527 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:24.469529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:24.469543 (+ 14us) server_negotiation.cc:407] Connection header received
0928 08:54:24.469612 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:24.469613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:24.469627 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:24.469676 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:24.469940 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:24.470435 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:24.471076 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:24.471342 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:24.471403 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:24.473936 (+ 2533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.195:59467: BlockingRecv error: recv got EOF from 127.23.167.195:59467 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 08:54:25 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:25 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:25.245394 960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:25.237805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.193:46029)
0928 08:54:25.237942 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:25.237954 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:25.238010 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:25.238134 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:25.238135 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:25.238291 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:25.238339 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:25.238341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:25.238957 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:25.238958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:25.239527 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:25.239529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:25.239826 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:25.239826 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:25.239861 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:25.243549 (+ 3688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:25.243554 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:25.245316 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
W20260928 08:54:25.245491 961 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:25.237895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:46029 (local address 127.23.167.254:38859)
0928 08:54:25.238002 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:25.238004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:25.238013 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:54:25.238058 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:25.238059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:25.238068 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:25.238125 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:25.238387 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:25.238889 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:25.239587 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:25.239778 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:25.239822 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:25.245415 (+ 5593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.193:46029: BlockingRecv error: recv got EOF from 127.23.167.193:46029 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":44,"threads_started":1}
Sep 28 08:54:25 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:25 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:25.356833 962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:25.352434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.194:47227)
0928 08:54:25.352573 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:25.352586 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:25.352652 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:25.352795 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:25.352796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:25.352975 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:25.353022 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:25.353024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:25.353694 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:25.353695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:25.354261 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:25.354262 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:25.354592 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:25.354593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:25.354647 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:25.355091 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:25.355096 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:25.356777 (+ 1681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":67,"threads_started":1}
W20260928 08:54:25.356961 961 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:25.352520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:47227 (local address 127.23.167.254:38859)
0928 08:54:25.352577 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:25.352579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:25.352632 (+ 53us) server_negotiation.cc:407] Connection header received
0928 08:54:25.352720 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:25.352721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:25.352734 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:25.352780 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:25.353107 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:25.353619 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:25.354340 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:25.354535 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:25.354587 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:25.356870 (+ 2283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.194:47227: BlockingRecv error: recv got EOF from 127.23.167.194:47227 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 28 08:54:25 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:25 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:25.479032 963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:25.474394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.195:59023)
0928 08:54:25.474555 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:25.474567 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:25.474640 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:25.474789 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:25.474790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:25.474952 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:25.475002 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:25.475004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:25.475725 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:25.475727 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:25.476249 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:25.476251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:25.476601 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:25.476602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:25.476658 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:25.477118 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:25.477123 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:25.478974 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":76,"threads_started":1}
W20260928 08:54:25.479132 961 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:25.474553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:59023 (local address 127.23.167.254:38859)
0928 08:54:25.474614 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:25.474616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:25.474625 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:54:25.474704 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:25.474705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:25.474720 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:25.474768 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:25.475079 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:25.475643 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:25.476324 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:25.476539 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:25.476598 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:25.479067 (+ 2469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.195:59023: BlockingRecv error: recv got EOF from 127.23.167.195:59023 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 08:54:26 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:26 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:26.250480 964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:26.245892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.193:45257)
0928 08:54:26.246062 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:26.246072 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:26.246130 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:26.246256 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:26.246257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:26.246424 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:26.246479 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:26.246481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:26.247178 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:26.247179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:26.247809 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:26.247811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:26.248152 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:26.248153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:26.248210 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:26.248608 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:26.248612 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:26.250354 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":103,"threads_started":1}
W20260928 08:54:26.250527 965 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:26.246009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:45257 (local address 127.23.167.254:38859)
0928 08:54:26.246110 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:26.246112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:26.246121 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:54:26.246177 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:26.246178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:26.246188 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:26.246247 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:26.246539 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:26.247079 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:26.247881 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:26.248084 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:26.248137 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:26.250462 (+ 2325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.193:45257: BlockingRecv error: recv got EOF from 127.23.167.193:45257 (error 108)
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":49,"threads_started":1}
Sep 28 08:54:26 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:26 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:26.361882 966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:26.357280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.194:33963)
0928 08:54:26.357483 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:26.357495 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:26.357562 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:26.357683 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:26.357684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:26.357868 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:26.357914 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:26.357915 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:26.358566 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:26.358567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:26.359117 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:26.359119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:26.359475 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:26.359476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:26.359533 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:26.359984 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:26.359990 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:26.361808 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":89,"threads_started":1}
W20260928 08:54:26.362030 965 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:26.357364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:33963 (local address 127.23.167.254:38859)
0928 08:54:26.357433 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:26.357435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:26.357555 (+ 120us) server_negotiation.cc:407] Connection header received
0928 08:54:26.357616 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:26.357617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:26.357629 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:26.357678 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:26.357998 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:26.358493 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:26.359199 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:26.359409 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:26.359466 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:26.361961 (+ 2495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.194:33963: BlockingRecv error: recv got EOF from 127.23.167.194:33963 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 28 08:54:26 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:26 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:26.483646 967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:26.479485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.195:37351)
0928 08:54:26.479618 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:26.479628 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:26.479691 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:26.479809 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:26.479810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:26.479969 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:26.480015 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:26.480017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:26.480621 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:26.480622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:26.481195 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:26.481198 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:26.481482 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:26.481483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:26.481522 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:26.481920 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:26.481923 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:26.483577 (+ 1654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":65,"threads_started":1}
W20260928 08:54:26.483737 965 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:26.479540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:37351 (local address 127.23.167.254:38859)
0928 08:54:26.479590 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:26.479592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:26.479683 (+ 91us) server_negotiation.cc:407] Connection header received
0928 08:54:26.479745 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:26.479746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:26.479758 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:26.479795 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:26.480063 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:26.480565 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:26.481250 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:26.481429 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:26.481473 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:26.483673 (+ 2200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.195:37351: BlockingRecv error: recv got EOF from 127.23.167.195:37351 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 28 08:54:27 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:27 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:27.255908 968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:27.251080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.193:37839)
0928 08:54:27.251303 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:27.251314 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:27.251388 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:27.251539 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:27.251539 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:27.251701 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:27.251749 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:27.251751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:27.252373 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:27.252374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:27.252968 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:27.252971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:27.253375 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:27.253376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:27.253417 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:27.253972 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:27.253977 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:27.255847 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":157,"threads_started":1}
W20260928 08:54:27.256055 969 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:27.251213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:37839 (local address 127.23.167.254:38859)
0928 08:54:27.251328 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:27.251330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:27.251391 (+ 61us) server_negotiation.cc:407] Connection header received
0928 08:54:27.251458 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:27.251459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:27.251472 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:27.251525 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:27.251807 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:27.252316 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:27.253092 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:27.253307 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:27.253381 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:27.255974 (+ 2593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.193:37839: BlockingRecv error: recv got EOF from 127.23.167.193:37839 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
Sep 28 08:54:27 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:27 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:27.366840 970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:27.362352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.194:51955)
0928 08:54:27.362496 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:27.362508 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:27.362576 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:27.362703 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:27.362704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:27.362880 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:27.362930 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:27.362939 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:27.363640 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:27.363641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:27.364204 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:27.364206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:27.364542 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:27.364543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:27.364600 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:27.365049 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:27.365054 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:27.366772 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":58,"threads_started":1}
W20260928 08:54:27.366942 969 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:27.362458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:51955 (local address 127.23.167.254:38859)
0928 08:54:27.362525 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:27.362527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:27.362558 (+ 31us) server_negotiation.cc:407] Connection header received
0928 08:54:27.362640 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:27.362640 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:27.362652 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:27.362698 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:27.363015 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:27.363559 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:27.364284 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:27.364483 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:27.364537 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:27.366880 (+ 2343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.194:51955: BlockingRecv error: recv got EOF from 127.23.167.194:51955 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 08:54:27 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:27 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:27.488273 971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:27.484102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.195:52235)
0928 08:54:27.484232 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:27.484244 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:27.484305 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:27.484427 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:27.484427 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:27.484588 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:27.484635 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:27.484636 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:27.485305 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:27.485307 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:27.485863 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:27.485865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:27.486169 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:27.486169 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:27.486207 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:27.486605 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:27.486609 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:27.488205 (+ 1596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
W20260928 08:54:27.488374 969 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:27.484211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:52235 (local address 127.23.167.254:38859)
0928 08:54:27.484289 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:27.484291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:27.484301 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:54:27.484350 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:27.484351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:27.484363 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:27.484415 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:27.484687 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:27.485227 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:27.485931 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:27.486118 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:27.486163 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:27.488305 (+ 2142us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.195:52235: BlockingRecv error: recv got EOF from 127.23.167.195:52235 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 08:54:28 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:28 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:28.261037 972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:28.256419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.193:45237)
0928 08:54:28.256612 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:28.256622 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:28.256674 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:28.256807 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:28.256807 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:28.256972 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:28.257019 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:28.257021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:28.257682 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:28.257684 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:28.258230 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:28.258232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:28.258552 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:28.258553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:28.258590 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:28.259028 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:28.259032 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:28.260979 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":115,"threads_started":1}
W20260928 08:54:28.261168 973 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:28.256539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:45237 (local address 127.23.167.254:38859)
0928 08:54:28.256673 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:28.256675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:28.256685 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:54:28.256731 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:28.256732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:28.256742 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:28.256796 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:28.257086 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:28.257606 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:28.258294 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:28.258499 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:28.258547 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:28.261099 (+ 2552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.193:45237: BlockingRecv error: recv got EOF from 127.23.167.193:45237 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
Sep 28 08:54:28 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:28 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:28.372162 974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:28.367379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.194:53217)
0928 08:54:28.367568 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:28.367582 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:28.367660 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:28.367791 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:28.367791 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:28.367976 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:28.368027 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:28.368028 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:28.368871 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:28.368872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:28.369442 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:28.369445 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:28.369777 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:28.369778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:28.369840 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:28.370308 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:28.370313 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:28.372083 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":87,"threads_started":1}
W20260928 08:54:28.372239 973 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:28.367548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:53217 (local address 127.23.167.254:38859)
0928 08:54:28.367609 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:28.367611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:28.367641 (+ 30us) server_negotiation.cc:407] Connection header received
0928 08:54:28.367711 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:28.367711 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:28.367722 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:28.367776 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:28.368107 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:28.368802 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:28.369518 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:28.369713 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:28.369767 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:28.372185 (+ 2418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.194:53217: BlockingRecv error: recv got EOF from 127.23.167.194:53217 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 28 08:54:28 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:28 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:28.493403 975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:28.488744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.195:56001)
0928 08:54:28.488893 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:28.488905 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:28.488971 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:28.489119 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:28.489120 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:28.489282 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:28.489329 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:28.489331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:28.489951 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:28.489952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:28.490507 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:28.490510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:28.490839 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:28.490839 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:28.490874 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:28.491343 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:28.491347 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:28.493330 (+ 1983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":64,"threads_started":1}
W20260928 08:54:28.493499 973 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:28.488844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:56001 (local address 127.23.167.254:38859)
0928 08:54:28.488890 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:28.488893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:28.488954 (+ 61us) server_negotiation.cc:407] Connection header received
0928 08:54:28.489013 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:28.489014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:28.489027 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:28.489106 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:28.489380 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:28.489870 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:28.490585 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:28.490782 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:28.490834 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:28.493435 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.195:56001: BlockingRecv error: recv got EOF from 127.23.167.195:56001 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 08:54:29 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:29 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:29.265986 976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:29.261460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.193:46841)
0928 08:54:29.261605 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:29.261615 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:29.261661 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:29.261802 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:29.261802 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:29.261964 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:29.262014 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:29.262016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:29.262634 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:29.262635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:29.263178 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:29.263179 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:29.263533 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:29.263535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:29.263579 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:29.264028 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:29.264032 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:29.265915 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":72,"threads_started":1}
W20260928 08:54:29.266108 977 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:29.261569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:46841 (local address 127.23.167.254:38859)
0928 08:54:29.261686 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:29.261688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:29.261696 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:54:29.261719 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:29.261720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:29.261731 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:29.261790 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:29.262073 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:29.262577 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:29.263242 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:29.263466 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:29.263519 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:29.266030 (+ 2511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.193:46841: BlockingRecv error: recv got EOF from 127.23.167.193:46841 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":52,"threads_started":1}
Sep 28 08:54:29 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:29 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:29.377144 978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:29.372677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.194:38079)
0928 08:54:29.372806 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:29.372818 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:29.372886 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:29.373049 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:29.373050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:29.373249 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:29.373297 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:29.373299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:29.373986 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:29.373987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:29.374512 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:29.374514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:29.374855 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:29.374856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:29.374911 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:29.375344 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:29.375347 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:29.377044 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":78,"threads_started":1}
W20260928 08:54:29.377279 977 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:29.372781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:38079 (local address 127.23.167.254:38859)
0928 08:54:29.372830 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:29.372832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:29.372868 (+ 36us) server_negotiation.cc:407] Connection header received
0928 08:54:29.372969 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:29.372970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:29.372983 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:29.373032 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:29.373371 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:29.373903 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:29.374586 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:29.374794 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:29.374849 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:29.377209 (+ 2360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.194:38079: BlockingRecv error: recv got EOF from 127.23.167.194:38079 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 08:54:29 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:29 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.195@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:29.498163 979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:29.493855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.195:51009)
0928 08:54:29.493991 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:29.494002 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:29.494060 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:29.494178 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:29.494178 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:29.494334 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:29.494380 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:29.494381 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:29.495020 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:29.495021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:29.495595 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:29.495597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:29.495902 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:29.495902 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:29.495939 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:29.496402 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:29.496408 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:29.498099 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":101,"threads_started":1}
W20260928 08:54:29.498243 977 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:29.493904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:51009 (local address 127.23.167.254:38859)
0928 08:54:29.493955 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:29.493957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:29.494053 (+ 96us) server_negotiation.cc:407] Connection header received
0928 08:54:29.494103 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:29.494103 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:29.494117 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:29.494168 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:29.494435 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:29.494947 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:29.495661 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:29.495852 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:29.495899 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:29.498185 (+ 2286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.195:51009: BlockingRecv error: recv got EOF from 127.23.167.195:51009 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 28 08:54:30 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:30 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.193@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:30.275915 980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:30.266614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.193:43659)
0928 08:54:30.266734 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:30.266747 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:30.266812 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:30.266989 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:30.266990 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:30.267204 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:30.267263 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:30.267266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:30.267938 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:30.267940 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:30.268473 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:30.268474 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:30.268810 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:30.268811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:30.268857 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:30.274050 (+ 5193us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:30.274055 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:30.275851 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":70,"threads_started":1}
W20260928 08:54:30.276042 981 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:30.266697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:43659 (local address 127.23.167.254:38859)
0928 08:54:30.266813 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:30.266814 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:30.266863 (+ 49us) server_negotiation.cc:407] Connection header received
0928 08:54:30.266887 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:30.266887 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:30.266902 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:30.266962 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:30.267335 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:30.267844 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:30.268552 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:30.268753 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:30.268803 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:30.275971 (+ 7168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.193:43659: BlockingRecv error: recv got EOF from 127.23.167.193:43659 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
Sep 28 08:54:30 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:54:30 dist-test-slave-pbjp krb5kdc[32391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.167.194@KRBTEST.COM for oryx/127.23.167.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:54:30.381903 982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:30.377589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38859 (local address 127.23.167.194:52981)
0928 08:54:30.377726 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:30.377738 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:30.377799 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:30.377917 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:30.377918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:30.378066 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:30.378114 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:30.378115 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:30.378792 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:30.378793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:30.379353 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:30.379355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:30.379690 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:30.379690 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:30.379726 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:30.380171 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:30.380176 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:30.381836 (+ 1660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.167.254:38859: Server oryx/127.23.167.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
W20260928 08:54:30.381990 981 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:30.377650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:52981 (local address 127.23.167.254:38859)
0928 08:54:30.377707 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:30.377709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:30.377792 (+ 83us) server_negotiation.cc:407] Connection header received
0928 08:54:30.377850 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:30.377850 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:30.377861 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:30.377907 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:30.378200 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:30.378722 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:30.379438 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:30.379635 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:30.379688 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:30.381912 (+ 2224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.167.194:52981: BlockingRecv error: recv got EOF from 127.23.167.194:52981 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 08:54:30.417635 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 506
I20260928 08:54:30.421870 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 658
I20260928 08:54:30.425717 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 790
I20260928 08:54:30.429498 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 436
2026-09-28T08:54:30Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17295 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T08:54:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:54:30Z Disabled control of system clock
I20260928 08:54:30.455780 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:37633
--webserver_interface=127.23.167.254
--webserver_port=0
--builtin_ntp_servers=127.23.167.212:43351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:37633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 08:54:30.534562 986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:54:30.534765 986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:54:30.534786 986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:54:30.536080 986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:54:30.536124 986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:54:30.536137 986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:54:30.536149 986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:54:30.536163 986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:54:30.537590 986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:43351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:37633
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:37633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:54:30.537833 986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:54:30.538148 986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:54:30.543462 994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:30.543438 991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:30.543496 986 server_base.cc:1034] running on GCE node
W20260928 08:54:30.543438 992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:30.543900 986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:54:30.544179 986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:54:30.545324 986 hybrid_clock.cc:648] HybridClock initialized: now 1790585670545309 us; error 36 us; skew 500 ppm
I20260928 08:54:30.546578 986 webserver.cc:460] Webserver started at http://127.23.167.254:40963/ using document root <none> and password file <none>
I20260928 08:54:30.546767 986 fs_manager.cc:359] Metadata directory not provided
I20260928 08:54:30.546808 986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:54:30.546909 986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:54:30.547971 986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "1157ff4d2633439ca591076b8043b4fc"
format_stamp: "Formatted at 2026-09-28 08:54:30 on dist-test-slave-pbjp"
server_key: "55ee50cb82c5cc6ae613e492a401447e"
server_key_iv: "a7537bb97fa2b2c0437fd12227ea9b3e"
server_key_version: "encryptionkey@0"
I20260928 08:54:30.548261 986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "1157ff4d2633439ca591076b8043b4fc"
format_stamp: "Formatted at 2026-09-28 08:54:30 on dist-test-slave-pbjp"
server_key: "55ee50cb82c5cc6ae613e492a401447e"
server_key_iv: "a7537bb97fa2b2c0437fd12227ea9b3e"
server_key_version: "encryptionkey@0"
I20260928 08:54:30.549482 986 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:54:30.550105 1000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:30.550253 986 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:54:30.550330 986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "1157ff4d2633439ca591076b8043b4fc"
format_stamp: "Formatted at 2026-09-28 08:54:30 on dist-test-slave-pbjp"
server_key: "55ee50cb82c5cc6ae613e492a401447e"
server_key_iv: "a7537bb97fa2b2c0437fd12227ea9b3e"
server_key_version: "encryptionkey@0"
I20260928 08:54:30.550384 986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:54:30.559607 986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:54:30.559875 986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:54:30.559984 986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:54:30.563542 986 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:37633
I20260928 08:54:30.563593 1052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:37633 every 8 connection(s)
I20260928 08:54:30.563896 986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:54:30.564604 1053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:54:30.567219 1053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1157ff4d2633439ca591076b8043b4fc: Bootstrap starting.
I20260928 08:54:30.567773 1053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1157ff4d2633439ca591076b8043b4fc: Neither blocks nor log segments found. Creating new log.
I20260928 08:54:30.568027 1053 log.cc:824] T 00000000000000000000000000000000 P 1157ff4d2633439ca591076b8043b4fc: Log is configured to *not* fsync() on all Append() calls
I20260928 08:54:30.568675 1053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1157ff4d2633439ca591076b8043b4fc: No bootstrap required, opened a new log
I20260928 08:54:30.569634 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 986
I20260928 08:54:30.569702 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:54:30.569860 24223 external_mini_cluster.cc:1428] Setting key 7fc47ae1a8efe640cc39ceb88e2b6e54
I20260928 08:54:30.570425 1053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1157ff4d2633439ca591076b8043b4fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1157ff4d2633439ca591076b8043b4fc" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 37633 } }
I20260928 08:54:30.570601 1053 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1157ff4d2633439ca591076b8043b4fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:54:30.570638 1053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1157ff4d2633439ca591076b8043b4fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1157ff4d2633439ca591076b8043b4fc, State: Initialized, Role: FOLLOWER
I20260928 08:54:30.570765 1053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1157ff4d2633439ca591076b8043b4fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1157ff4d2633439ca591076b8043b4fc" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 37633 } }
I20260928 08:54:30.570838 1053 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1157ff4d2633439ca591076b8043b4fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:54:30.570868 1053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1157ff4d2633439ca591076b8043b4fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:54:30.570936 1053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1157ff4d2633439ca591076b8043b4fc [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:54:30.571506 1053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1157ff4d2633439ca591076b8043b4fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1157ff4d2633439ca591076b8043b4fc" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 37633 } }
I20260928 08:54:30.571593 1053 leader_election.cc:302] T 00000000000000000000000000000000 P 1157ff4d2633439ca591076b8043b4fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1157ff4d2633439ca591076b8043b4fc; no voters:
I20260928 08:54:30.571736 1053 leader_election.cc:288] T 00000000000000000000000000000000 P 1157ff4d2633439ca591076b8043b4fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:54:30.571766 1058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1157ff4d2633439ca591076b8043b4fc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:54:30.571905 1058 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1157ff4d2633439ca591076b8043b4fc [term 1 LEADER]: Becoming Leader. State: Replica: 1157ff4d2633439ca591076b8043b4fc, State: Running, Role: LEADER
I20260928 08:54:30.571985 1053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1157ff4d2633439ca591076b8043b4fc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:54:30.572019 1058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1157ff4d2633439ca591076b8043b4fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1157ff4d2633439ca591076b8043b4fc" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 37633 } }
I20260928 08:54:30.572388 1058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1157ff4d2633439ca591076b8043b4fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1157ff4d2633439ca591076b8043b4fc. Latest consensus state: current_term: 1 leader_uuid: "1157ff4d2633439ca591076b8043b4fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1157ff4d2633439ca591076b8043b4fc" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 37633 } } }
I20260928 08:54:30.572397 1060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1157ff4d2633439ca591076b8043b4fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1157ff4d2633439ca591076b8043b4fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1157ff4d2633439ca591076b8043b4fc" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 37633 } } }
I20260928 08:54:30.572510 1058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1157ff4d2633439ca591076b8043b4fc [sys.catalog]: This master's current role is: LEADER
I20260928 08:54:30.572520 1060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1157ff4d2633439ca591076b8043b4fc [sys.catalog]: This master's current role is: LEADER
I20260928 08:54:30.573128 1064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:54:30.573666 1064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:54:30.575404 1064 catalog_manager.cc:1347] Generated new cluster ID: 7b9e09ac03ec4e9d85fac22b1b717025
I20260928 08:54:30.575503 1064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:54:30.575984 1057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:30.570364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41460 (local address 127.23.167.254:37633)
0928 08:54:30.570588 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:30.570596 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:30.570619 (+ 23us) server_negotiation.cc:407] Connection header received
0928 08:54:30.570734 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:30.570740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:30.570814 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:30.570953 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:30.571169 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:30.572688 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:30.573904 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:30.574374 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:30.574489 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:30.575173 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:54:30.575178 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:54:30.575178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:54:30.575375 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:54:30.575418 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:54:30.575434 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:54:30.575497 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:30.575703 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 08:54:30.575779 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":108,"threads_started":1}
I20260928 08:54:30.588155 1064 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:54:30.588724 1064 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:54:30.593756 1064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1157ff4d2633439ca591076b8043b4fc: Generated new TSK 0
I20260928 08:54:30.594009 1064 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:54:30.602895 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:0
--local_ip_for_outbound_sockets=127.23.167.193
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:37633
--builtin_ntp_servers=127.23.167.212:43351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 08:54:30.681901 1077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:54:30.682096 1077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:54:30.682123 1077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:54:30.683249 1077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:54:30.683287 1077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:54:30.683318 1077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:54:30.684623 1077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:43351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:37633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:54:30.684839 1077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:54:30.685114 1077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:54:30.689779 1082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:30.689832 1077 server_base.cc:1034] running on GCE node
W20260928 08:54:30.689798 1083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:30.689978 1085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:30.690250 1077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:54:30.690481 1077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:54:30.691630 1077 hybrid_clock.cc:648] HybridClock initialized: now 1790585670691615 us; error 29 us; skew 500 ppm
I20260928 08:54:30.692800 1077 webserver.cc:460] Webserver started at http://127.23.167.193:46629/ using document root <none> and password file <none>
I20260928 08:54:30.693015 1077 fs_manager.cc:359] Metadata directory not provided
I20260928 08:54:30.693084 1077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:54:30.693214 1077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:54:30.694270 1077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/data/instance:
uuid: "f47d0c6fa62740129ef68ed075c37b44"
format_stamp: "Formatted at 2026-09-28 08:54:30 on dist-test-slave-pbjp"
server_key: "6209ed9d4df997bfb918c1e60d31e4b4"
server_key_iv: "47ea3988e368bd1f427dd4baa51a6c28"
server_key_version: "encryptionkey@0"
I20260928 08:54:30.694625 1077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance:
uuid: "f47d0c6fa62740129ef68ed075c37b44"
format_stamp: "Formatted at 2026-09-28 08:54:30 on dist-test-slave-pbjp"
server_key: "6209ed9d4df997bfb918c1e60d31e4b4"
server_key_iv: "47ea3988e368bd1f427dd4baa51a6c28"
server_key_version: "encryptionkey@0"
I20260928 08:54:30.695785 1077 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 08:54:30.696403 1091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:30.696557 1077 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:54:30.696635 1077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "f47d0c6fa62740129ef68ed075c37b44"
format_stamp: "Formatted at 2026-09-28 08:54:30 on dist-test-slave-pbjp"
server_key: "6209ed9d4df997bfb918c1e60d31e4b4"
server_key_iv: "47ea3988e368bd1f427dd4baa51a6c28"
server_key_version: "encryptionkey@0"
I20260928 08:54:30.696702 1077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:54:30.729352 1077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:54:30.729694 1077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:54:30.729835 1077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:54:30.730064 1077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:54:30.730405 1077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:54:30.730440 1077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:30.730481 1077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:54:30.730505 1077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:30.736251 1077 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:46287
I20260928 08:54:30.736328 1204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:46287 every 8 connection(s)
I20260928 08:54:30.736670 1077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
I20260928 08:54:30.737857 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 1077
I20260928 08:54:30.737927 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance
I20260928 08:54:30.738106 24223 external_mini_cluster.cc:1428] Setting key 4823c7b767d3bd959332ebcc271bce9e
I20260928 08:54:30.738934 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:0
--local_ip_for_outbound_sockets=127.23.167.194
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:37633
--builtin_ntp_servers=127.23.167.212:43351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 08:54:30.742110 1057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:30.737406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:58381 (local address 127.23.167.254:37633)
0928 08:54:30.737481 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:30.737484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:30.737725 (+ 241us) server_negotiation.cc:407] Connection header received
0928 08:54:30.738337 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:30.738338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:30.738351 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:30.738393 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:30.739107 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:30.739725 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:30.740723 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:30.740912 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:30.740976 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:30.741876 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:54:30.741877 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:54:30.741877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:54:30.741881 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:54:30.741897 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:54:30.741902 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:54:30.741939 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:30.742021 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 08:54:30.742050 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 08:54:30.742364 1207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:30.737386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:37633 (local address 127.23.167.193:58381)
0928 08:54:30.737577 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:30.737613 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:30.738181 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:30.738508 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:30.738512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:30.738521 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:30.739003 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:30.739008 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:30.739825 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:30.739825 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:30.740638 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:30.740640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:30.740994 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:30.740995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:30.741178 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:30.741711 (+ 533us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:54:30.741711 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:30.741751 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:54:30.741782 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:54:30.741950 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:54:30.741954 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:54:30.741958 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:54:30.742024 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 08:54:30.742139 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
I20260928 08:54:30.742884 1205 heartbeater.cc:347] Connected to a master server at 127.23.167.254:37633
I20260928 08:54:30.742985 1205 heartbeater.cc:464] Registering TS with master...
I20260928 08:54:30.743166 1205 heartbeater.cc:511] Master 127.23.167.254:37633 requested a full tablet report, sending...
I20260928 08:54:30.743657 1017 ts_manager.cc:194] Registered new tserver with Master: f47d0c6fa62740129ef68ed075c37b44 (127.23.167.193:46287)
I20260928 08:54:30.744233 1017 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.167.193:58381
W20260928 08:54:30.818902 1208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:54:30.819115 1208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:54:30.819144 1208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:54:30.820302 1208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:54:30.820344 1208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:54:30.820377 1208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:54:30.821695 1208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:43351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:37633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:54:30.821946 1208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:54:30.822239 1208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 08:54:30.826977 1208 server_base.cc:1034] running on GCE node
W20260928 08:54:30.826962 1213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:30.827049 1214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:30.827004 1216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:30.827394 1208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:54:30.827625 1208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:54:30.828790 1208 hybrid_clock.cc:648] HybridClock initialized: now 1790585670828776 us; error 26 us; skew 500 ppm
I20260928 08:54:30.830049 1208 webserver.cc:460] Webserver started at http://127.23.167.194:34347/ using document root <none> and password file <none>
I20260928 08:54:30.830281 1208 fs_manager.cc:359] Metadata directory not provided
I20260928 08:54:30.830327 1208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:54:30.830458 1208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:54:30.831634 1208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/data/instance:
uuid: "7821ba4e574e4a999ccb417f727223bc"
format_stamp: "Formatted at 2026-09-28 08:54:30 on dist-test-slave-pbjp"
server_key: "b190d249d9d861c96199d47df11151ac"
server_key_iv: "ecb22f130cc5fc72aa2dd804eda69dce"
server_key_version: "encryptionkey@0"
I20260928 08:54:30.832015 1208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance:
uuid: "7821ba4e574e4a999ccb417f727223bc"
format_stamp: "Formatted at 2026-09-28 08:54:30 on dist-test-slave-pbjp"
server_key: "b190d249d9d861c96199d47df11151ac"
server_key_iv: "ecb22f130cc5fc72aa2dd804eda69dce"
server_key_version: "encryptionkey@0"
I20260928 08:54:30.833302 1208 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:54:30.833977 1222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:30.834124 1208 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:54:30.834203 1208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "7821ba4e574e4a999ccb417f727223bc"
format_stamp: "Formatted at 2026-09-28 08:54:30 on dist-test-slave-pbjp"
server_key: "b190d249d9d861c96199d47df11151ac"
server_key_iv: "ecb22f130cc5fc72aa2dd804eda69dce"
server_key_version: "encryptionkey@0"
I20260928 08:54:30.834270 1208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:54:30.858047 1208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:54:30.858355 1208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:54:30.858489 1208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:54:30.858704 1208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:54:30.859035 1208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:54:30.859067 1208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:30.859097 1208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:54:30.859112 1208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:30.864737 1208 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:42855
I20260928 08:54:30.864799 1335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:42855 every 8 connection(s)
I20260928 08:54:30.865185 1208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
I20260928 08:54:30.871025 1057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:30.865918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:40607 (local address 127.23.167.254:37633)
0928 08:54:30.865984 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:30.865986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:30.866221 (+ 235us) server_negotiation.cc:407] Connection header received
0928 08:54:30.866813 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:30.866815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:30.866832 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:30.866881 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:30.867597 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:30.868094 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:30.869380 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:30.869642 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:30.869704 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:30.870713 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:54:30.870715 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:54:30.870716 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:54:30.870720 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:54:30.870741 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:54:30.870747 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:54:30.870818 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:30.870897 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 08:54:30.870933 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 08:54:30.871236 1338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:30.865904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:37633 (local address 127.23.167.194:40607)
0928 08:54:30.866098 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:30.866132 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:30.866692 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:30.866971 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:30.866976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:30.866987 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:30.867500 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:30.867506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:30.868181 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:30.868182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:30.869266 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:30.869270 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:30.869777 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:30.869778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:30.869935 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:30.870565 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:54:30.870566 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:30.870596 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:54:30.870620 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:54:30.870820 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:54:30.870824 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:54:30.870827 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:54:30.870887 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 08:54:30.871001 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":88,"threads_started":1}
I20260928 08:54:30.871759 1336 heartbeater.cc:347] Connected to a master server at 127.23.167.254:37633
I20260928 08:54:30.871868 1336 heartbeater.cc:464] Registering TS with master...
I20260928 08:54:30.872079 1336 heartbeater.cc:511] Master 127.23.167.254:37633 requested a full tablet report, sending...
I20260928 08:54:30.872466 1017 ts_manager.cc:194] Registered new tserver with Master: 7821ba4e574e4a999ccb417f727223bc (127.23.167.194:42855)
I20260928 08:54:30.872944 1017 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.167.194:40607
I20260928 08:54:30.874325 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 1208
I20260928 08:54:30.874419 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance
I20260928 08:54:30.874615 24223 external_mini_cluster.cc:1428] Setting key 9bbaf863f3f24be34bb3fe57db3b7b86
I20260928 08:54:30.875571 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:0
--local_ip_for_outbound_sockets=127.23.167.195
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:37633
--builtin_ntp_servers=127.23.167.212:43351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 08:54:30.956022 1339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:54:30.956228 1339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:54:30.956247 1339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:54:30.957803 1339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:54:30.957854 1339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:54:30.957885 1339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:54:30.959528 1339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:43351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:37633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:54:30.959774 1339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:54:30.960080 1339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:54:30.964798 1344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:30.964882 1339 server_base.cc:1034] running on GCE node
W20260928 08:54:30.964810 1345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:30.964810 1347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:30.965196 1339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:54:30.965430 1339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:54:30.966578 1339 hybrid_clock.cc:648] HybridClock initialized: now 1790585670966564 us; error 31 us; skew 500 ppm
I20260928 08:54:30.967825 1339 webserver.cc:460] Webserver started at http://127.23.167.195:44407/ using document root <none> and password file <none>
I20260928 08:54:30.968062 1339 fs_manager.cc:359] Metadata directory not provided
I20260928 08:54:30.968112 1339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:54:30.968237 1339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:54:30.969317 1339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/data/instance:
uuid: "366a31ae518b46268358f06244674776"
format_stamp: "Formatted at 2026-09-28 08:54:30 on dist-test-slave-pbjp"
server_key: "389e9e5e6936f43ab537fb69d261b53f"
server_key_iv: "37a6aab10ebbfede27477d0fdcac7042"
server_key_version: "encryptionkey@0"
I20260928 08:54:30.969642 1339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance:
uuid: "366a31ae518b46268358f06244674776"
format_stamp: "Formatted at 2026-09-28 08:54:30 on dist-test-slave-pbjp"
server_key: "389e9e5e6936f43ab537fb69d261b53f"
server_key_iv: "37a6aab10ebbfede27477d0fdcac7042"
server_key_version: "encryptionkey@0"
I20260928 08:54:30.970799 1339 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:54:30.971446 1353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:30.971582 1339 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:54:30.971658 1339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "366a31ae518b46268358f06244674776"
format_stamp: "Formatted at 2026-09-28 08:54:30 on dist-test-slave-pbjp"
server_key: "389e9e5e6936f43ab537fb69d261b53f"
server_key_iv: "37a6aab10ebbfede27477d0fdcac7042"
server_key_version: "encryptionkey@0"
I20260928 08:54:30.971719 1339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:54:30.997756 1339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:54:30.998059 1339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:54:30.998183 1339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:54:30.998402 1339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:54:30.998733 1339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:54:30.998765 1339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:30.998793 1339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:54:30.998811 1339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:31.004148 1339 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:33141
I20260928 08:54:31.004195 1466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:33141 every 8 connection(s)
I20260928 08:54:31.004510 1339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
I20260928 08:54:31.009555 1057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:31.005221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:60161 (local address 127.23.167.254:37633)
0928 08:54:31.005277 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:31.005279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:31.005532 (+ 253us) server_negotiation.cc:407] Connection header received
0928 08:54:31.006025 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:31.006026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:31.006037 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:31.006085 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:31.006733 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:31.007222 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:31.008204 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:31.008387 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:31.008436 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:31.009289 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:54:31.009290 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:54:31.009290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:54:31.009294 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:54:31.009310 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:54:31.009314 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:54:31.009360 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:31.009456 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 08:54:31.009481 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 08:54:31.009756 1469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:31.005202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:37633 (local address 127.23.167.195:60161)
0928 08:54:31.005410 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:31.005445 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:31.005944 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:31.006151 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:31.006154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:31.006163 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:31.006663 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:31.006667 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:31.007302 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:31.007303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:31.008129 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:31.008131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:31.008491 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:31.008492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:31.008620 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:31.009142 (+ 522us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:54:31.009143 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:31.009181 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:54:31.009205 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:54:31.009393 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:54:31.009396 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:54:31.009399 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:54:31.009450 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 08:54:31.009542 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260928 08:54:31.010207 1467 heartbeater.cc:347] Connected to a master server at 127.23.167.254:37633
I20260928 08:54:31.010295 1467 heartbeater.cc:464] Registering TS with master...
I20260928 08:54:31.010455 1467 heartbeater.cc:511] Master 127.23.167.254:37633 requested a full tablet report, sending...
I20260928 08:54:31.010605 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 1339
I20260928 08:54:31.010686 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance
I20260928 08:54:31.010819 1017 ts_manager.cc:194] Registered new tserver with Master: 366a31ae518b46268358f06244674776 (127.23.167.195:33141)
I20260928 08:54:31.010874 24223 external_mini_cluster.cc:1428] Setting key 12b4b474431cde109f1dd143f84b9f15
I20260928 08:54:31.011277 1017 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.167.195:60161
I20260928 08:54:31.011785 24223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 08:54:31.013531 1057 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:31.013009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41474 (local address 127.23.167.254:37633)
0928 08:54:31.013100 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:31.013102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:31.013179 (+ 77us) server_negotiation.cc:407] Connection header received
0928 08:54:31.013254 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:31.013255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:31.013267 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:31.013330 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:31.013467 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41474: BlockingRecv error: recv got EOF from 127.0.0.1:41474 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 08:54:31.013623 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 1077
I20260928 08:54:31.018139 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 1208
I20260928 08:54:31.021893 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 1339
I20260928 08:54:31.025643 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 986
2026-09-28T08:54:31Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (594 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T08:54:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:54:31Z Disabled control of system clock
I20260928 08:54:31.050146 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:33425
--webserver_interface=127.23.167.254
--webserver_port=0
--builtin_ntp_servers=127.23.167.212:36681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:33425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 08:54:31.129932 1478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:54:31.130131 1478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:54:31.130152 1478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:54:31.131302 1478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:54:31.131347 1478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:54:31.131358 1478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:54:31.131376 1478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:54:31.131386 1478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:54:31.132733 1478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:36681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:33425
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:33425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:54:31.132975 1478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:54:31.133273 1478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:54:31.137748 1483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:31.137763 1486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:31.137763 1484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:31.138082 1478 server_base.cc:1034] running on GCE node
I20260928 08:54:31.138314 1478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:54:31.138576 1478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:54:31.139721 1478 hybrid_clock.cc:648] HybridClock initialized: now 1790585671139707 us; error 24 us; skew 500 ppm
I20260928 08:54:31.140836 1478 webserver.cc:460] Webserver started at http://127.23.167.254:45915/ using document root <none> and password file <none>
I20260928 08:54:31.141055 1478 fs_manager.cc:359] Metadata directory not provided
I20260928 08:54:31.141144 1478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:54:31.141268 1478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:54:31.142297 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "83cad8b5a5744aa6bac76353ed4a4e3f"
format_stamp: "Formatted at 2026-09-28 08:54:31 on dist-test-slave-pbjp"
server_key: "d65b75bf62fe3b0511997e0e84b2c8e9"
server_key_iv: "a2f57eaffe934245012e702eb5c61f0c"
server_key_version: "encryptionkey@0"
I20260928 08:54:31.142656 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "83cad8b5a5744aa6bac76353ed4a4e3f"
format_stamp: "Formatted at 2026-09-28 08:54:31 on dist-test-slave-pbjp"
server_key: "d65b75bf62fe3b0511997e0e84b2c8e9"
server_key_iv: "a2f57eaffe934245012e702eb5c61f0c"
server_key_version: "encryptionkey@0"
I20260928 08:54:31.143916 1478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 08:54:31.144590 1492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:31.144740 1478 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:54:31.144821 1478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "83cad8b5a5744aa6bac76353ed4a4e3f"
format_stamp: "Formatted at 2026-09-28 08:54:31 on dist-test-slave-pbjp"
server_key: "d65b75bf62fe3b0511997e0e84b2c8e9"
server_key_iv: "a2f57eaffe934245012e702eb5c61f0c"
server_key_version: "encryptionkey@0"
I20260928 08:54:31.144888 1478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:54:31.148172 1478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:54:31.148428 1478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:54:31.148552 1478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:54:31.152202 1478 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:33425
I20260928 08:54:31.152242 1544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:33425 every 8 connection(s)
I20260928 08:54:31.152550 1478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:54:31.153203 1545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:54:31.154105 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 1478
I20260928 08:54:31.154194 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:54:31.154359 24223 external_mini_cluster.cc:1428] Setting key fc715f9548d4112f3bb35424ae98e2c3
I20260928 08:54:31.156489 1545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83cad8b5a5744aa6bac76353ed4a4e3f: Bootstrap starting.
I20260928 08:54:31.157008 1548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:31.154951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43400 (local address 127.23.167.254:33425)
0928 08:54:31.155163 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:31.155173 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:31.155206 (+ 33us) server_negotiation.cc:407] Connection header received
0928 08:54:31.155733 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:31.155742 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:31.155811 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:31.155962 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:31.156569 (+ 607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:54:31.156573 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:54:31.156573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:54:31.156577 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:54:31.156624 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:54:31.156637 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:54:31.156699 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:31.156758 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 08:54:31.156820 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
I20260928 08:54:31.157225 1545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83cad8b5a5744aa6bac76353ed4a4e3f: Neither blocks nor log segments found. Creating new log.
I20260928 08:54:31.157506 1545 log.cc:824] T 00000000000000000000000000000000 P 83cad8b5a5744aa6bac76353ed4a4e3f: Log is configured to *not* fsync() on all Append() calls
I20260928 08:54:31.158175 1545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83cad8b5a5744aa6bac76353ed4a4e3f: No bootstrap required, opened a new log
I20260928 08:54:31.159482 1545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83cad8b5a5744aa6bac76353ed4a4e3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83cad8b5a5744aa6bac76353ed4a4e3f" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 33425 } }
I20260928 08:54:31.159626 1545 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83cad8b5a5744aa6bac76353ed4a4e3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:54:31.159654 1545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83cad8b5a5744aa6bac76353ed4a4e3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83cad8b5a5744aa6bac76353ed4a4e3f, State: Initialized, Role: FOLLOWER
I20260928 08:54:31.159744 1545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83cad8b5a5744aa6bac76353ed4a4e3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83cad8b5a5744aa6bac76353ed4a4e3f" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 33425 } }
I20260928 08:54:31.159793 1545 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83cad8b5a5744aa6bac76353ed4a4e3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:54:31.159811 1545 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83cad8b5a5744aa6bac76353ed4a4e3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:54:31.159852 1545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83cad8b5a5744aa6bac76353ed4a4e3f [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:54:31.160774 1545 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83cad8b5a5744aa6bac76353ed4a4e3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83cad8b5a5744aa6bac76353ed4a4e3f" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 33425 } }
I20260928 08:54:31.160871 1545 leader_election.cc:302] T 00000000000000000000000000000000 P 83cad8b5a5744aa6bac76353ed4a4e3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83cad8b5a5744aa6bac76353ed4a4e3f; no voters:
I20260928 08:54:31.160987 1545 leader_election.cc:288] T 00000000000000000000000000000000 P 83cad8b5a5744aa6bac76353ed4a4e3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:54:31.161031 1550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83cad8b5a5744aa6bac76353ed4a4e3f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:54:31.161216 1550 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83cad8b5a5744aa6bac76353ed4a4e3f [term 1 LEADER]: Becoming Leader. State: Replica: 83cad8b5a5744aa6bac76353ed4a4e3f, State: Running, Role: LEADER
I20260928 08:54:31.161257 1545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83cad8b5a5744aa6bac76353ed4a4e3f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:54:31.161314 1550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83cad8b5a5744aa6bac76353ed4a4e3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83cad8b5a5744aa6bac76353ed4a4e3f" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 33425 } }
I20260928 08:54:31.161689 1552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83cad8b5a5744aa6bac76353ed4a4e3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83cad8b5a5744aa6bac76353ed4a4e3f. Latest consensus state: current_term: 1 leader_uuid: "83cad8b5a5744aa6bac76353ed4a4e3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83cad8b5a5744aa6bac76353ed4a4e3f" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 33425 } } }
I20260928 08:54:31.161791 1552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83cad8b5a5744aa6bac76353ed4a4e3f [sys.catalog]: This master's current role is: LEADER
I20260928 08:54:31.161695 1551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83cad8b5a5744aa6bac76353ed4a4e3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83cad8b5a5744aa6bac76353ed4a4e3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83cad8b5a5744aa6bac76353ed4a4e3f" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 33425 } } }
I20260928 08:54:31.161924 1551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83cad8b5a5744aa6bac76353ed4a4e3f [sys.catalog]: This master's current role is: LEADER
I20260928 08:54:31.162060 1559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:54:31.162518 1559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:54:31.163959 1559 catalog_manager.cc:1347] Generated new cluster ID: 2c231f4aa5a24a11b3867b4ecae8c31a
I20260928 08:54:31.164002 1559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:54:31.172530 1559 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:54:31.172709 1559 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:54:31.182860 1559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83cad8b5a5744aa6bac76353ed4a4e3f: Generated new TSK 0
I20260928 08:54:31.183104 1559 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:54:31.191578 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:0
--local_ip_for_outbound_sockets=127.23.167.193
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:33425
--builtin_ntp_servers=127.23.167.212:36681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 08:54:31.272415 1569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:54:31.272662 1569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:54:31.272693 1569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:54:31.273957 1569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:54:31.274000 1569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:54:31.274034 1569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:54:31.275388 1569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:36681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:33425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:54:31.275642 1569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:54:31.275947 1569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:54:31.280735 1574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:31.280826 1577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:31.280737 1575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:31.281096 1569 server_base.cc:1034] running on GCE node
I20260928 08:54:31.281287 1569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:54:31.281548 1569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:54:31.282718 1569 hybrid_clock.cc:648] HybridClock initialized: now 1790585671282701 us; error 33 us; skew 500 ppm
I20260928 08:54:31.284159 1569 webserver.cc:460] Webserver started at http://127.23.167.193:35501/ using document root <none> and password file <none>
I20260928 08:54:31.284421 1569 fs_manager.cc:359] Metadata directory not provided
I20260928 08:54:31.284479 1569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:54:31.284605 1569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:54:31.286029 1569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/data/instance:
uuid: "11327b25c8bc498c886494fe695bc15c"
format_stamp: "Formatted at 2026-09-28 08:54:31 on dist-test-slave-pbjp"
server_key: "dff2c331a5d5cf64fa2a05666a7a3783"
server_key_iv: "f3ab0e075bb67c633ca904d6f8385ab9"
server_key_version: "encryptionkey@0"
I20260928 08:54:31.286451 1569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance:
uuid: "11327b25c8bc498c886494fe695bc15c"
format_stamp: "Formatted at 2026-09-28 08:54:31 on dist-test-slave-pbjp"
server_key: "dff2c331a5d5cf64fa2a05666a7a3783"
server_key_iv: "f3ab0e075bb67c633ca904d6f8385ab9"
server_key_version: "encryptionkey@0"
I20260928 08:54:31.287781 1569 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:54:31.288451 1583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:31.288606 1569 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:54:31.288677 1569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "11327b25c8bc498c886494fe695bc15c"
format_stamp: "Formatted at 2026-09-28 08:54:31 on dist-test-slave-pbjp"
server_key: "dff2c331a5d5cf64fa2a05666a7a3783"
server_key_iv: "f3ab0e075bb67c633ca904d6f8385ab9"
server_key_version: "encryptionkey@0"
I20260928 08:54:31.288744 1569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:54:31.291963 1569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:54:31.292198 1569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:54:31.292310 1569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:54:31.292496 1569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:54:31.292776 1569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:54:31.292806 1569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:31.292825 1569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:54:31.292850 1569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:31.298135 1569 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:42425
I20260928 08:54:31.298203 1696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:42425 every 8 connection(s)
I20260928 08:54:31.298472 1569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
I20260928 08:54:31.301692 1548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:31.299230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:36007 (local address 127.23.167.254:33425)
0928 08:54:31.299279 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:31.299281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:31.299566 (+ 285us) server_negotiation.cc:407] Connection header received
0928 08:54:31.300060 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:31.300061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:31.300069 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:31.300116 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:31.301435 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:54:31.301436 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:54:31.301437 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:54:31.301442 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:54:31.301465 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:54:31.301469 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:54:31.301525 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:31.301609 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 08:54:31.301623 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 08:54:31.301889 1699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:31.299223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:33425 (local address 127.23.167.193:36007)
0928 08:54:31.299440 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:31.299483 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:31.299962 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:31.300187 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:31.300191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:31.300201 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:31.301247 (+ 1046us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:54:31.301247 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:31.301286 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:54:31.301338 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:54:31.301543 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:54:31.301549 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:54:31.301552 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:54:31.301621 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 08:54:31.301683 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":92,"threads_started":1}
I20260928 08:54:31.303035 1697 heartbeater.cc:347] Connected to a master server at 127.23.167.254:33425
I20260928 08:54:31.303129 1697 heartbeater.cc:464] Registering TS with master...
I20260928 08:54:31.303277 1697 heartbeater.cc:511] Master 127.23.167.254:33425 requested a full tablet report, sending...
I20260928 08:54:31.303874 1509 ts_manager.cc:194] Registered new tserver with Master: 11327b25c8bc498c886494fe695bc15c (127.23.167.193:42425)
I20260928 08:54:31.306315 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 1569
I20260928 08:54:31.306406 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance
I20260928 08:54:31.306679 24223 external_mini_cluster.cc:1428] Setting key f5d8e91b8fffe54ed0002f4c40501da9
I20260928 08:54:31.307768 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:0
--local_ip_for_outbound_sockets=127.23.167.194
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:33425
--builtin_ntp_servers=127.23.167.212:36681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 08:54:31.386718 1700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:54:31.386914 1700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:54:31.386942 1700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:54:31.388053 1700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:54:31.388089 1700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:54:31.388123 1700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:54:31.389421 1700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:36681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:33425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:54:31.389657 1700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:54:31.389921 1700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:54:31.394637 1705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:31.394683 1706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:31.394677 1708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:31.394912 1700 server_base.cc:1034] running on GCE node
I20260928 08:54:31.395112 1700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:54:31.395300 1700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:54:31.396437 1700 hybrid_clock.cc:648] HybridClock initialized: now 1790585671396408 us; error 44 us; skew 500 ppm
I20260928 08:54:31.397657 1700 webserver.cc:460] Webserver started at http://127.23.167.194:40687/ using document root <none> and password file <none>
I20260928 08:54:31.397876 1700 fs_manager.cc:359] Metadata directory not provided
I20260928 08:54:31.397933 1700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:54:31.398058 1700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:54:31.399087 1700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/data/instance:
uuid: "438f2fb81480417ba51ecc8d84e37847"
format_stamp: "Formatted at 2026-09-28 08:54:31 on dist-test-slave-pbjp"
server_key: "98165bdf31e8dfb9209d8670124d6dde"
server_key_iv: "1e42245ee65c7dd3db480425a774a432"
server_key_version: "encryptionkey@0"
I20260928 08:54:31.399413 1700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance:
uuid: "438f2fb81480417ba51ecc8d84e37847"
format_stamp: "Formatted at 2026-09-28 08:54:31 on dist-test-slave-pbjp"
server_key: "98165bdf31e8dfb9209d8670124d6dde"
server_key_iv: "1e42245ee65c7dd3db480425a774a432"
server_key_version: "encryptionkey@0"
I20260928 08:54:31.400588 1700 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:54:31.401321 1714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:31.401480 1700 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:54:31.401557 1700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "438f2fb81480417ba51ecc8d84e37847"
format_stamp: "Formatted at 2026-09-28 08:54:31 on dist-test-slave-pbjp"
server_key: "98165bdf31e8dfb9209d8670124d6dde"
server_key_iv: "1e42245ee65c7dd3db480425a774a432"
server_key_version: "encryptionkey@0"
I20260928 08:54:31.401624 1700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:54:31.404881 1700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:54:31.405172 1700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:54:31.405293 1700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:54:31.405484 1700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:54:31.405761 1700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:54:31.405792 1700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:31.405819 1700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:54:31.405843 1700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:31.411231 1700 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:36861
I20260928 08:54:31.411269 1827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:36861 every 8 connection(s)
I20260928 08:54:31.411587 1700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
I20260928 08:54:31.412021 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 1700
I20260928 08:54:31.412103 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance
I20260928 08:54:31.412309 24223 external_mini_cluster.cc:1428] Setting key b23c71f51bc2f5930ab7ac5a386747f4
I20260928 08:54:31.413219 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:0
--local_ip_for_outbound_sockets=127.23.167.195
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:33425
--builtin_ntp_servers=127.23.167.212:36681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 08:54:31.414810 1548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:31.412247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:37085 (local address 127.23.167.254:33425)
0928 08:54:31.412299 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:31.412301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:31.412672 (+ 371us) server_negotiation.cc:407] Connection header received
0928 08:54:31.413269 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:31.413270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:31.413276 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:31.413325 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:31.414581 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:54:31.414583 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:54:31.414583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:54:31.414587 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:54:31.414602 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:54:31.414606 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:54:31.414657 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:31.414743 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 08:54:31.414754 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 08:54:31.415030 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:31.412255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:33425 (local address 127.23.167.194:37085)
0928 08:54:31.412564 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:31.412606 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:31.413180 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:31.413403 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:31.413407 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:31.413417 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:31.414301 (+ 884us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:54:31.414301 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:31.414393 (+ 92us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:54:31.414435 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:54:31.414674 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:54:31.414678 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:54:31.414681 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:54:31.414752 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 08:54:31.414818 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":75,"threads_started":1}
I20260928 08:54:31.415443 1828 heartbeater.cc:347] Connected to a master server at 127.23.167.254:33425
I20260928 08:54:31.415536 1828 heartbeater.cc:464] Registering TS with master...
I20260928 08:54:31.415676 1828 heartbeater.cc:511] Master 127.23.167.254:33425 requested a full tablet report, sending...
I20260928 08:54:31.416010 1509 ts_manager.cc:194] Registered new tserver with Master: 438f2fb81480417ba51ecc8d84e37847 (127.23.167.194:36861)
W20260928 08:54:31.492189 1831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:54:31.492396 1831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:54:31.492426 1831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:54:31.493608 1831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:54:31.493644 1831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:54:31.493669 1831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:54:31.494984 1831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:36681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:33425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:54:31.495180 1831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:54:31.495419 1831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:54:31.500348 1836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:31.500396 1839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:31.500411 1837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:31.500532 1831 server_base.cc:1034] running on GCE node
I20260928 08:54:31.500736 1831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:54:31.500957 1831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:54:31.502095 1831 hybrid_clock.cc:648] HybridClock initialized: now 1790585671502078 us; error 30 us; skew 500 ppm
I20260928 08:54:31.503255 1831 webserver.cc:460] Webserver started at http://127.23.167.195:34777/ using document root <none> and password file <none>
I20260928 08:54:31.503499 1831 fs_manager.cc:359] Metadata directory not provided
I20260928 08:54:31.503547 1831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:54:31.503665 1831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:54:31.504704 1831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/data/instance:
uuid: "1b55cb774a2749a5bad332c50fda3882"
format_stamp: "Formatted at 2026-09-28 08:54:31 on dist-test-slave-pbjp"
server_key: "4070eb624043ff90b284122f1d996f83"
server_key_iv: "f9e2511e00ec42380e5ad81d26cd43f1"
server_key_version: "encryptionkey@0"
I20260928 08:54:31.505045 1831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance:
uuid: "1b55cb774a2749a5bad332c50fda3882"
format_stamp: "Formatted at 2026-09-28 08:54:31 on dist-test-slave-pbjp"
server_key: "4070eb624043ff90b284122f1d996f83"
server_key_iv: "f9e2511e00ec42380e5ad81d26cd43f1"
server_key_version: "encryptionkey@0"
I20260928 08:54:31.506258 1831 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:54:31.506910 1845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:31.507069 1831 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:54:31.507146 1831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "1b55cb774a2749a5bad332c50fda3882"
format_stamp: "Formatted at 2026-09-28 08:54:31 on dist-test-slave-pbjp"
server_key: "4070eb624043ff90b284122f1d996f83"
server_key_iv: "f9e2511e00ec42380e5ad81d26cd43f1"
server_key_version: "encryptionkey@0"
I20260928 08:54:31.507213 1831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:54:31.510460 1831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:54:31.510699 1831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:54:31.510807 1831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:54:31.511001 1831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:54:31.511274 1831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:54:31.511304 1831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:31.511340 1831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:54:31.511364 1831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:31.516564 1831 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:39055
I20260928 08:54:31.516602 1958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:39055 every 8 connection(s)
I20260928 08:54:31.516897 1831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
I20260928 08:54:31.517894 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 1831
I20260928 08:54:31.517979 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance
I20260928 08:54:31.518149 24223 external_mini_cluster.cc:1428] Setting key 6a5ac1486a69d5ba98ae380537b345a9
I20260928 08:54:31.519743 1548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:31.517595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:42527 (local address 127.23.167.254:33425)
0928 08:54:31.517651 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:31.517653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:31.517879 (+ 226us) server_negotiation.cc:407] Connection header received
0928 08:54:31.518379 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:31.518382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:31.518389 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:31.518452 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:31.519571 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:54:31.519572 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:54:31.519572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:54:31.519576 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:54:31.519590 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:54:31.519594 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:54:31.519635 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:31.519681 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 08:54:31.519691 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 08:54:31.519913 1961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:31.517568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:33425 (local address 127.23.167.195:42527)
0928 08:54:31.517740 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:31.517780 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:31.518280 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:31.518523 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:31.518527 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:31.518537 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:31.519444 (+ 907us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:54:31.519444 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:31.519473 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:54:31.519501 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:54:31.519635 (+ 134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:54:31.519638 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:54:31.519641 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:54:31.519687 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 08:54:31.519733 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
I20260928 08:54:31.520308 1959 heartbeater.cc:347] Connected to a master server at 127.23.167.254:33425
I20260928 08:54:31.520390 1959 heartbeater.cc:464] Registering TS with master...
I20260928 08:54:31.520524 1959 heartbeater.cc:511] Master 127.23.167.254:33425 requested a full tablet report, sending...
I20260928 08:54:31.520833 1509 ts_manager.cc:194] Registered new tserver with Master: 1b55cb774a2749a5bad332c50fda3882 (127.23.167.195:39055)
I20260928 08:54:31.529760 24223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 08:54:31.531453 1548 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:54:31.531008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43408 (local address 127.23.167.254:33425)
0928 08:54:31.531058 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:31.531060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:31.531138 (+ 78us) server_negotiation.cc:407] Connection header received
0928 08:54:31.531194 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:31.531195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:31.531200 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:31.531234 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:31.531334 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43408: BlockingRecv error: recv got EOF from 127.0.0.1:43408 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 08:54:31.531507 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 1569
I20260928 08:54:31.535389 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 1700
I20260928 08:54:31.539048 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 1831
I20260928 08:54:31.542649 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 1478
2026-09-28T08:54:31Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (517 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:54:31 dist-test-slave-pbjp krb5kdc[1969](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:54:31 dist-test-slave-pbjp krb5kdc[1969](info): set up 2 sockets
Sep 28 08:54:31 dist-test-slave-pbjp krb5kdc[1969](info): commencing operation
krb5kdc: starting...
W20260928 08:54:32.496969 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:54:32 dist-test-slave-pbjp krb5kdc[1969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:54:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:54:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.167.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
I20260928 08:54:32.608343 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:38527
--webserver_interface=127.23.167.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:38527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:54:32.687860 1982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:54:32.688091 1982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:54:32.688122 1982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:54:32.689304 1982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:54:32.689343 1982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:54:32.689361 1982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:54:32.689378 1982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:54:32.689392 1982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:54:32.690917 1982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:38527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:38527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:54:32.691174 1982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:54:32.691383 1982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:54:32.695971 1988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:32.696053 1982 server_base.cc:1034] running on GCE node
W20260928 08:54:32.695990 1987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:32.696161 1990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:32.696410 1982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:54:32.696677 1982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:54:32.697834 1982 hybrid_clock.cc:648] HybridClock initialized: now 1790585672697816 us; error 27 us; skew 500 ppm
Sep 28 08:54:32 dist-test-slave-pbjp krb5kdc[1969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585672, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:54:32.700167 1982 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:54:32.700594 1982 webserver.cc:460] Webserver started at http://127.23.167.254:36481/ using document root <none> and password file <none>
I20260928 08:54:32.700807 1982 fs_manager.cc:359] Metadata directory not provided
I20260928 08:54:32.700853 1982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:54:32.700976 1982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:54:32.702028 1982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "3e633e5ffcd7497d89774f3b24b5f423"
format_stamp: "Formatted at 2026-09-28 08:54:32 on dist-test-slave-pbjp"
server_key: "f69e3db6889c042ffd544d8957657f49"
server_key_iv: "948e377171295bda2a7b585ef7f7490d"
server_key_version: "encryptionkey@0"
I20260928 08:54:32.702373 1982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "3e633e5ffcd7497d89774f3b24b5f423"
format_stamp: "Formatted at 2026-09-28 08:54:32 on dist-test-slave-pbjp"
server_key: "f69e3db6889c042ffd544d8957657f49"
server_key_iv: "948e377171295bda2a7b585ef7f7490d"
server_key_version: "encryptionkey@0"
I20260928 08:54:32.703593 1982 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 08:54:32.704325 1997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:32.704500 1982 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:54:32.704582 1982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "3e633e5ffcd7497d89774f3b24b5f423"
format_stamp: "Formatted at 2026-09-28 08:54:32 on dist-test-slave-pbjp"
server_key: "f69e3db6889c042ffd544d8957657f49"
server_key_iv: "948e377171295bda2a7b585ef7f7490d"
server_key_version: "encryptionkey@0"
I20260928 08:54:32.704649 1982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:54:32.716257 1982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:54:32.718472 1982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:54:32.718613 1982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:54:32.722402 1982 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:38527
I20260928 08:54:32.722450 2049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:38527 every 8 connection(s)
I20260928 08:54:32.722826 1982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:54:32.723791 2050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:54:32.726198 2050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e633e5ffcd7497d89774f3b24b5f423: Bootstrap starting.
I20260928 08:54:32.726768 2050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e633e5ffcd7497d89774f3b24b5f423: Neither blocks nor log segments found. Creating new log.
I20260928 08:54:32.727028 2050 log.cc:824] T 00000000000000000000000000000000 P 3e633e5ffcd7497d89774f3b24b5f423: Log is configured to *not* fsync() on all Append() calls
I20260928 08:54:32.727710 2050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e633e5ffcd7497d89774f3b24b5f423: No bootstrap required, opened a new log
I20260928 08:54:32.729324 2050 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e633e5ffcd7497d89774f3b24b5f423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e633e5ffcd7497d89774f3b24b5f423" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38527 } }
I20260928 08:54:32.729476 2050 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e633e5ffcd7497d89774f3b24b5f423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:54:32.729499 2050 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e633e5ffcd7497d89774f3b24b5f423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e633e5ffcd7497d89774f3b24b5f423, State: Initialized, Role: FOLLOWER
I20260928 08:54:32.729657 2050 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e633e5ffcd7497d89774f3b24b5f423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e633e5ffcd7497d89774f3b24b5f423" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38527 } }
I20260928 08:54:32.729727 2050 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e633e5ffcd7497d89774f3b24b5f423 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:54:32.729751 2050 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e633e5ffcd7497d89774f3b24b5f423 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:54:32.729800 2050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e633e5ffcd7497d89774f3b24b5f423 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:54:32.730618 2050 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e633e5ffcd7497d89774f3b24b5f423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e633e5ffcd7497d89774f3b24b5f423" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38527 } }
I20260928 08:54:32.730713 2050 leader_election.cc:302] T 00000000000000000000000000000000 P 3e633e5ffcd7497d89774f3b24b5f423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e633e5ffcd7497d89774f3b24b5f423; no voters:
I20260928 08:54:32.730868 2050 leader_election.cc:288] T 00000000000000000000000000000000 P 3e633e5ffcd7497d89774f3b24b5f423 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:54:32.730911 2053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e633e5ffcd7497d89774f3b24b5f423 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:54:32.731067 2053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e633e5ffcd7497d89774f3b24b5f423 [term 1 LEADER]: Becoming Leader. State: Replica: 3e633e5ffcd7497d89774f3b24b5f423, State: Running, Role: LEADER
I20260928 08:54:32.731125 2050 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e633e5ffcd7497d89774f3b24b5f423 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:54:32.731160 2053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e633e5ffcd7497d89774f3b24b5f423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e633e5ffcd7497d89774f3b24b5f423" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38527 } }
I20260928 08:54:32.731551 2054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e633e5ffcd7497d89774f3b24b5f423 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e633e5ffcd7497d89774f3b24b5f423" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e633e5ffcd7497d89774f3b24b5f423" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38527 } } }
I20260928 08:54:32.731566 2055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e633e5ffcd7497d89774f3b24b5f423 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e633e5ffcd7497d89774f3b24b5f423. Latest consensus state: current_term: 1 leader_uuid: "3e633e5ffcd7497d89774f3b24b5f423" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e633e5ffcd7497d89774f3b24b5f423" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38527 } } }
I20260928 08:54:32.731730 2055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e633e5ffcd7497d89774f3b24b5f423 [sys.catalog]: This master's current role is: LEADER
I20260928 08:54:32.731721 2054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e633e5ffcd7497d89774f3b24b5f423 [sys.catalog]: This master's current role is: LEADER
I20260928 08:54:32.732142 2060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:54:32.732688 2060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:54:32.732841 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 1982
I20260928 08:54:32.732937 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:54:32.733155 24223 external_mini_cluster.cc:1428] Setting key dcb4179ca2b62e05d77e67a37d4f5563
I20260928 08:54:32.734421 2060 catalog_manager.cc:1347] Generated new cluster ID: 5ee0e4970dc145febd4c5b995301fcca
I20260928 08:54:32.734469 2060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 08:54:32 dist-test-slave-pbjp krb5kdc[1969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:54:32.749397 2060 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:54:32.750011 2060 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:54:32.750986 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:32.733611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48180 (local address 127.23.167.254:38527)
0928 08:54:32.733768 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:32.733778 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:32.740317 (+ 6539us) server_negotiation.cc:407] Connection header received
0928 08:54:32.740470 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:32.740479 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:32.740558 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:32.740719 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:32.741441 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:32.742903 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:32.743887 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:32.744560 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:32.744663 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:32.747402 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:54:32.747407 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:54:32.747407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:54:32.747421 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:54:32.749309 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:32.749705 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:32.749708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:32.749709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:32.749831 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:32.750041 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:32.750041 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:32.750042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:32.750323 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:54:32.750482 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:32.750589 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 08:54:32.750742 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
I20260928 08:54:32.759656 2060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e633e5ffcd7497d89774f3b24b5f423: Generated new TSK 0
I20260928 08:54:32.759850 2060 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.167.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
I20260928 08:54:32.791292 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:0
--local_ip_for_outbound_sockets=127.23.167.193
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:38527
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:54:32.871778 2076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:54:32.871975 2076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:54:32.871996 2076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:54:32.873132 2076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:54:32.873167 2076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:54:32.873193 2076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:54:32.874656 2076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:38527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:54:32.874874 2076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:54:32.875083 2076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:54:32.879825 2081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:32.879987 2076 server_base.cc:1034] running on GCE node
W20260928 08:54:32.879837 2084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:32.879843 2082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:32.880259 2076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:54:32.880466 2076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:54:32.881631 2076 hybrid_clock.cc:648] HybridClock initialized: now 1790585672881616 us; error 27 us; skew 500 ppm
Sep 28 08:54:32 dist-test-slave-pbjp krb5kdc[1969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585672, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:54:32.883970 2076 init.cc:377] Logged in from keytab as kudu/127.23.167.193@KRBTEST.COM (short username kudu)
I20260928 08:54:32.884363 2076 webserver.cc:460] Webserver started at http://127.23.167.193:37725/ using document root <none> and password file <none>
I20260928 08:54:32.884549 2076 fs_manager.cc:359] Metadata directory not provided
I20260928 08:54:32.884598 2076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:54:32.884716 2076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:54:32.885766 2076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/data/instance:
uuid: "aec300b90f174e159db1453af76be440"
format_stamp: "Formatted at 2026-09-28 08:54:32 on dist-test-slave-pbjp"
server_key: "575bc0741b07d6b38a245dcc1ece7027"
server_key_iv: "7f6b857829c67085ae5361109ab76cdd"
server_key_version: "encryptionkey@0"
I20260928 08:54:32.886088 2076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance:
uuid: "aec300b90f174e159db1453af76be440"
format_stamp: "Formatted at 2026-09-28 08:54:32 on dist-test-slave-pbjp"
server_key: "575bc0741b07d6b38a245dcc1ece7027"
server_key_iv: "7f6b857829c67085ae5361109ab76cdd"
server_key_version: "encryptionkey@0"
I20260928 08:54:32.887279 2076 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:54:32.887941 2091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:32.888098 2076 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:54:32.888176 2076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "aec300b90f174e159db1453af76be440"
format_stamp: "Formatted at 2026-09-28 08:54:32 on dist-test-slave-pbjp"
server_key: "575bc0741b07d6b38a245dcc1ece7027"
server_key_iv: "7f6b857829c67085ae5361109ab76cdd"
server_key_version: "encryptionkey@0"
I20260928 08:54:32.888234 2076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:54:32.907829 2076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:54:32.910070 2076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:54:32.910197 2076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:54:32.910423 2076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:54:32.910758 2076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:54:32.910791 2076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:32.910821 2076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:54:32.910844 2076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:32.916492 2076 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:37739
I20260928 08:54:32.916548 2204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:37739 every 8 connection(s)
I20260928 08:54:32.916847 2076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
Sep 28 08:54:32 dist-test-slave-pbjp krb5kdc[1969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585672, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:54:32.925249 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:32.917578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:44419 (local address 127.23.167.254:38527)
0928 08:54:32.917637 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:32.917639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:32.917825 (+ 186us) server_negotiation.cc:407] Connection header received
0928 08:54:32.918301 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:32.918302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:32.918311 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:32.918373 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:32.919272 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:32.919782 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:32.920747 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:32.920937 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:32.920985 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:32.922939 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:54:32.922941 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:54:32.922941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:54:32.922948 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:54:32.924193 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:32.924533 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:32.924534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:32.924534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:32.924580 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:32.924749 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:32.924749 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:32.924749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:32.924842 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:54:32.924887 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:32.925112 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 08:54:32.925184 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 08:54:32.925419 2207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:32.917522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38527 (local address 127.23.167.193:44419)
0928 08:54:32.917697 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:32.917740 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:32.918221 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:32.918433 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:32.918443 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:32.918994 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:32.919199 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:32.919205 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:32.919849 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:32.919849 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:32.920683 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:32.920685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:32.921012 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:32.921013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:32.921160 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:32.921586 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:32.921594 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:32.922852 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:54:32.924254 (+ 1402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:54:32.924257 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:54:32.924259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:54:32.924479 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:54:32.924623 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:54:32.924623 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:54:32.924624 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:54:32.924710 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:54:32.924882 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:54:32.924885 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:54:32.924994 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 08:54:32.925088 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 08:54:32.925213 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":90,"threads_started":1}
I20260928 08:54:32.925894 2205 heartbeater.cc:347] Connected to a master server at 127.23.167.254:38527
I20260928 08:54:32.925993 2205 heartbeater.cc:464] Registering TS with master...
I20260928 08:54:32.926043 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 2076
I20260928 08:54:32.926124 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance
I20260928 08:54:32.926184 2205 heartbeater.cc:511] Master 127.23.167.254:38527 requested a full tablet report, sending...
I20260928 08:54:32.926321 24223 external_mini_cluster.cc:1428] Setting key 7d71ea5e312dfc99a00e77e634e45a0d
I20260928 08:54:32.926692 2014 ts_manager.cc:194] Registered new tserver with Master: aec300b90f174e159db1453af76be440 (127.23.167.193:37739)
I20260928 08:54:32.927337 2014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.193@KRBTEST.COM'} at 127.23.167.193:44419
WARNING: no policy specified for kudu/127.23.167.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
I20260928 08:54:32.949445 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:0
--local_ip_for_outbound_sockets=127.23.167.194
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:38527
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:54:33.029748 2210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:54:33.029947 2210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:54:33.029968 2210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:54:33.031095 2210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:54:33.031136 2210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:54:33.031162 2210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:54:33.032618 2210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:38527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:54:33.032832 2210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:54:33.033051 2210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:54:33.038272 2216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:33.038298 2218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:33.038300 2210 server_base.cc:1034] running on GCE node
W20260928 08:54:33.038272 2215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:33.038727 2210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:54:33.038977 2210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:54:33.040138 2210 hybrid_clock.cc:648] HybridClock initialized: now 1790585673040118 us; error 34 us; skew 500 ppm
Sep 28 08:54:33 dist-test-slave-pbjp krb5kdc[1969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585673, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:54:33.042598 2210 init.cc:377] Logged in from keytab as kudu/127.23.167.194@KRBTEST.COM (short username kudu)
I20260928 08:54:33.042986 2210 webserver.cc:460] Webserver started at http://127.23.167.194:45843/ using document root <none> and password file <none>
I20260928 08:54:33.043164 2210 fs_manager.cc:359] Metadata directory not provided
I20260928 08:54:33.043217 2210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:54:33.043341 2210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:54:33.044379 2210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/data/instance:
uuid: "ab3d1803f22146a99f38e3d02631abc2"
format_stamp: "Formatted at 2026-09-28 08:54:33 on dist-test-slave-pbjp"
server_key: "2b01f6fe5affe48f4aaa2de1c8d5351f"
server_key_iv: "5149c800834bc786997e5f79ec994cdf"
server_key_version: "encryptionkey@0"
I20260928 08:54:33.044700 2210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance:
uuid: "ab3d1803f22146a99f38e3d02631abc2"
format_stamp: "Formatted at 2026-09-28 08:54:33 on dist-test-slave-pbjp"
server_key: "2b01f6fe5affe48f4aaa2de1c8d5351f"
server_key_iv: "5149c800834bc786997e5f79ec994cdf"
server_key_version: "encryptionkey@0"
I20260928 08:54:33.045996 2210 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 08:54:33.046700 2225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:33.046888 2210 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:33.046973 2210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "ab3d1803f22146a99f38e3d02631abc2"
format_stamp: "Formatted at 2026-09-28 08:54:33 on dist-test-slave-pbjp"
server_key: "2b01f6fe5affe48f4aaa2de1c8d5351f"
server_key_iv: "5149c800834bc786997e5f79ec994cdf"
server_key_version: "encryptionkey@0"
I20260928 08:54:33.047029 2210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:54:33.057013 2210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:54:33.058974 2210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:54:33.059084 2210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:54:33.059267 2210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:54:33.059540 2210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:54:33.059571 2210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:33.059592 2210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:54:33.059607 2210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:33.065534 2210 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:34773
I20260928 08:54:33.065610 2338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:34773 every 8 connection(s)
I20260928 08:54:33.065887 2210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
Sep 28 08:54:33 dist-test-slave-pbjp krb5kdc[1969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585673, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:54:33.073794 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 2210
I20260928 08:54:33.073897 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance
I20260928 08:54:33.074093 24223 external_mini_cluster.cc:1428] Setting key 012bdcd470d5cea5608007cbe2ff1f35
I20260928 08:54:33.075098 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:33.066593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:55635 (local address 127.23.167.254:38527)
0928 08:54:33.066641 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:33.066643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:33.066820 (+ 177us) server_negotiation.cc:407] Connection header received
0928 08:54:33.067335 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:33.067337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:33.067350 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:33.067398 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:33.068371 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:33.068876 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:33.069913 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:33.070111 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:33.070167 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:33.072225 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:54:33.072226 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:54:33.072227 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:54:33.072234 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:54:33.073556 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:33.074048 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:33.074050 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:33.074050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:33.074110 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:33.074374 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:33.074375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:33.074375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:33.074552 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:54:33.074620 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:33.074918 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 08:54:33.075011 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 08:54:33.075189 2341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:33.066560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38527 (local address 127.23.167.194:55635)
0928 08:54:33.066694 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:33.066736 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:33.067233 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:33.067513 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:33.067517 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:33.068067 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:33.068264 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:33.068270 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:33.068960 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:33.068961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:33.069825 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:33.069828 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:33.070181 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:33.070182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:33.070312 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:33.070793 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:33.070802 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:33.072135 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:54:33.073631 (+ 1496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:54:33.073639 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:54:33.073641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:54:33.073930 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:54:33.074176 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:54:33.074177 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:54:33.074177 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:54:33.074300 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:54:33.074621 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:54:33.074627 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:54:33.074785 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 08:54:33.074880 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 08:54:33.075009 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":59,"threads_started":1}
I20260928 08:54:33.075601 2339 heartbeater.cc:347] Connected to a master server at 127.23.167.254:38527
I20260928 08:54:33.075690 2339 heartbeater.cc:464] Registering TS with master...
I20260928 08:54:33.075872 2339 heartbeater.cc:511] Master 127.23.167.254:38527 requested a full tablet report, sending...
I20260928 08:54:33.076201 2014 ts_manager.cc:194] Registered new tserver with Master: ab3d1803f22146a99f38e3d02631abc2 (127.23.167.194:34773)
I20260928 08:54:33.076637 2014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.194@KRBTEST.COM'} at 127.23.167.194:55635
WARNING: no policy specified for kudu/127.23.167.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
I20260928 08:54:33.099530 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:0
--local_ip_for_outbound_sockets=127.23.167.195
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:38527
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:54:33.178386 2344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:54:33.178599 2344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:54:33.178632 2344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:54:33.179819 2344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:54:33.179863 2344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:54:33.179898 2344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:54:33.181385 2344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:38527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:54:33.181624 2344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:54:33.181835 2344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:54:33.186690 2350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:33.186784 2344 server_base.cc:1034] running on GCE node
W20260928 08:54:33.186681 2352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:54:33.186681 2349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:54:33.187054 2344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:54:33.187294 2344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:54:33.188438 2344 hybrid_clock.cc:648] HybridClock initialized: now 1790585673188435 us; error 40 us; skew 500 ppm
Sep 28 08:54:33 dist-test-slave-pbjp krb5kdc[1969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585673, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:54:33.191032 2344 init.cc:377] Logged in from keytab as kudu/127.23.167.195@KRBTEST.COM (short username kudu)
I20260928 08:54:33.191496 2344 webserver.cc:460] Webserver started at http://127.23.167.195:44543/ using document root <none> and password file <none>
I20260928 08:54:33.191681 2344 fs_manager.cc:359] Metadata directory not provided
I20260928 08:54:33.191720 2344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:54:33.191810 2344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:54:33.192915 2344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/data/instance:
uuid: "4f57de7b8dbd45d3b27577bcd4a1da46"
format_stamp: "Formatted at 2026-09-28 08:54:33 on dist-test-slave-pbjp"
server_key: "e1b58413ea8dd87c872f7f6bc21fbd17"
server_key_iv: "aa96b2d1359786d4cf3814f761baee06"
server_key_version: "encryptionkey@0"
I20260928 08:54:33.193274 2344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance:
uuid: "4f57de7b8dbd45d3b27577bcd4a1da46"
format_stamp: "Formatted at 2026-09-28 08:54:33 on dist-test-slave-pbjp"
server_key: "e1b58413ea8dd87c872f7f6bc21fbd17"
server_key_iv: "aa96b2d1359786d4cf3814f761baee06"
server_key_version: "encryptionkey@0"
I20260928 08:54:33.194600 2344 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:54:33.195309 2359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:33.195443 2344 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:33.195542 2344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "4f57de7b8dbd45d3b27577bcd4a1da46"
format_stamp: "Formatted at 2026-09-28 08:54:33 on dist-test-slave-pbjp"
server_key: "e1b58413ea8dd87c872f7f6bc21fbd17"
server_key_iv: "aa96b2d1359786d4cf3814f761baee06"
server_key_version: "encryptionkey@0"
I20260928 08:54:33.195614 2344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:54:33.209610 2344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:54:33.211762 2344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:54:33.211885 2344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:54:33.212107 2344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:54:33.212400 2344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:54:33.212432 2344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:33.212463 2344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:54:33.212487 2344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:54:33.218189 2344 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:34103
I20260928 08:54:33.218238 2472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:34103 every 8 connection(s)
I20260928 08:54:33.218549 2344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
I20260928 08:54:33.224237 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 2344
I20260928 08:54:33.224337 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance
I20260928 08:54:33.224520 24223 external_mini_cluster.cc:1428] Setting key cb9fae39c0a7f256ad055541e835973d
Sep 28 08:54:33 dist-test-slave-pbjp krb5kdc[1969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585673, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:54:33.227806 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:33.219291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:46267 (local address 127.23.167.254:38527)
0928 08:54:33.219379 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:33.219382 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:33.219548 (+ 166us) server_negotiation.cc:407] Connection header received
0928 08:54:33.220063 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:33.220065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:33.220077 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:33.220130 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:33.221129 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:33.221629 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:33.222608 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:33.222800 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:33.222853 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:33.225117 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:54:33.225118 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:54:33.225118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:54:33.225125 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:54:33.226364 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:33.226834 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:33.226835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:33.226836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:33.226896 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:33.227169 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:33.227170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:33.227170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:33.227273 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:54:33.227322 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:33.227675 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 08:54:33.227740 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 08:54:33.228011 2475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:33.219224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38527 (local address 127.23.167.195:46267)
0928 08:54:33.219424 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:33.219469 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:33.219966 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:33.220202 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:33.220206 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:33.220804 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:54:33.221011 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:33.221016 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:33.221714 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:33.221715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:33.222525 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:33.222527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:33.222869 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:33.222870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:33.222987 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:33.223507 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:54:33.223517 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:54:33.224977 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:54:33.226510 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:54:33.226517 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:54:33.226519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:54:33.226745 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:54:33.226986 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:54:33.226987 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:54:33.226987 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:54:33.227111 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:54:33.227409 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:54:33.227416 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:54:33.227568 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 08:54:33.227660 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 08:54:33.227786 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":91,"threads_started":1}
I20260928 08:54:33.228416 2473 heartbeater.cc:347] Connected to a master server at 127.23.167.254:38527
I20260928 08:54:33.228505 2473 heartbeater.cc:464] Registering TS with master...
I20260928 08:54:33.228685 2473 heartbeater.cc:511] Master 127.23.167.254:38527 requested a full tablet report, sending...
I20260928 08:54:33.229035 2014 ts_manager.cc:194] Registered new tserver with Master: 4f57de7b8dbd45d3b27577bcd4a1da46 (127.23.167.195:34103)
I20260928 08:54:33.229409 2014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.195@KRBTEST.COM'} at 127.23.167.195:46267
I20260928 08:54:33.236254 24223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:54:33.243158 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:33.237483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48192 (local address 127.23.167.254:38527)
0928 08:54:33.237544 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:33.237546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:33.237608 (+ 62us) server_negotiation.cc:407] Connection header received
0928 08:54:33.237673 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:33.237674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:33.237684 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:33.237723 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:54:33.238269 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:33.238753 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:33.239475 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:33.239663 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:33.239715 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:33.240816 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:54:33.240817 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:54:33.240818 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:54:33.240825 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:54:33.242120 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:33.242424 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:33.242425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:33.242426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:33.242478 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:54:33.242638 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:54:33.242639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:54:33.242640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:54:33.242784 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:54:33.242871 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:33.242998 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 08:54:33.243068 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 08:54:33.244656 2014 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:54:33.246239 2014 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:54:33.252647 2486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:33.248633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50326 (local address 127.23.167.194:34773)
0928 08:54:33.248805 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:33.248809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:33.248817 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:54:33.248863 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:33.248864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:33.248878 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:33.249139 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:54:33.249212 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:33.250029 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:33.251542 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:33.252207 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:33.252269 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:33.252315 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:33.252514 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 08:54:33.252559 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":91,"threads_started":1}
I20260928 08:54:33.252660 2481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:33.248213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:34103 (local address 127.0.0.1:45008)
0928 08:54:33.248376 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:33.248416 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:33.248479 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:33.248765 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:33.248769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:33.248771 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:54:33.248834 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:33.248836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:33.250241 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:33.250242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:33.251348 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:33.251350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:33.252274 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:33.252276 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:33.252382 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:33.252400 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 08:54:33.252486 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 08:54:33.252554 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":38,"thread_start_us":58,"threads_started":1}
I20260928 08:54:33.252924 2484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:33.248358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45008 (local address 127.23.167.195:34103)
0928 08:54:33.248520 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:33.248525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:33.248536 (+ 11us) server_negotiation.cc:407] Connection header received
0928 08:54:33.248570 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:33.248573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:33.248590 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:33.248673 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:54:33.248909 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:33.250137 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:33.251425 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:33.252165 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:33.252251 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:33.252316 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:33.252801 (+ 485us) server_negotiation.cc:299] Negotiation successful
0928 08:54:33.252846 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":68,"threads_started":1}
I20260928 08:54:33.252660 2485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:33.248400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:34773 (local address 127.0.0.1:50326)
0928 08:54:33.248616 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:33.248624 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:33.248660 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:33.248969 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:33.248970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:33.248989 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:54:33.249036 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:33.249038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:33.250125 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:33.250126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:33.251454 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:33.251458 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:33.252286 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:33.252287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:33.252385 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:33.252400 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 08:54:33.252486 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 08:54:33.252555 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":79,"thread_start_us":51,"threads_started":1}
I20260928 08:54:33.253399 2482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:33.248213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:37739 (local address 127.0.0.1:39138)
0928 08:54:33.248580 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:33.248588 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:33.248624 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:33.248928 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:33.248928 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:33.248930 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:54:33.248987 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:33.248989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:33.250293 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:33.250295 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:33.251461 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:33.251463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:33.253147 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:33.253148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:33.253211 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:33.253215 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:54:33.253277 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 08:54:33.253328 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":5,"thread_start_us":58,"threads_started":1}
I20260928 08:54:33.253544 2483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:33.248298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39138 (local address 127.23.167.193:37739)
0928 08:54:33.248782 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:33.248788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:33.248797 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:54:33.248840 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:33.248842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:33.248858 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:33.248936 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:54:33.249103 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:33.250178 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:33.252354 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:33.253035 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:33.253146 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:33.253209 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:33.253353 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 08:54:33.253400 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":104,"threads_started":1}
I20260928 08:54:33.253782 2273 tablet_service.cc:1467] Processing CreateTablet for tablet 1ce1610623954041bc98ab3f4e5113f1 (DEFAULT_TABLE table=test-table [id=df0e870abd924394abbd2b4f49f010cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:54:33.253779 2407 tablet_service.cc:1467] Processing CreateTablet for tablet 1ce1610623954041bc98ab3f4e5113f1 (DEFAULT_TABLE table=test-table [id=df0e870abd924394abbd2b4f49f010cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:54:33.254151 2273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ce1610623954041bc98ab3f4e5113f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:54:33.254151 2407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ce1610623954041bc98ab3f4e5113f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:54:33.254096 2139 tablet_service.cc:1467] Processing CreateTablet for tablet 1ce1610623954041bc98ab3f4e5113f1 (DEFAULT_TABLE table=test-table [id=df0e870abd924394abbd2b4f49f010cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:54:33.254336 2139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ce1610623954041bc98ab3f4e5113f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:54:33.256863 2487 tablet_bootstrap.cc:492] T 1ce1610623954041bc98ab3f4e5113f1 P aec300b90f174e159db1453af76be440: Bootstrap starting.
I20260928 08:54:33.257441 2488 tablet_bootstrap.cc:492] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46: Bootstrap starting.
I20260928 08:54:33.257521 2487 tablet_bootstrap.cc:654] T 1ce1610623954041bc98ab3f4e5113f1 P aec300b90f174e159db1453af76be440: Neither blocks nor log segments found. Creating new log.
I20260928 08:54:33.258086 2488 tablet_bootstrap.cc:654] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46: Neither blocks nor log segments found. Creating new log.
I20260928 08:54:33.258086 2487 log.cc:824] T 1ce1610623954041bc98ab3f4e5113f1 P aec300b90f174e159db1453af76be440: Log is configured to *not* fsync() on all Append() calls
I20260928 08:54:33.258599 2488 log.cc:824] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46: Log is configured to *not* fsync() on all Append() calls
I20260928 08:54:33.258858 2487 tablet_bootstrap.cc:492] T 1ce1610623954041bc98ab3f4e5113f1 P aec300b90f174e159db1453af76be440: No bootstrap required, opened a new log
I20260928 08:54:33.258927 2487 ts_tablet_manager.cc:1397] T 1ce1610623954041bc98ab3f4e5113f1 P aec300b90f174e159db1453af76be440: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:54:33.258929 2489 tablet_bootstrap.cc:492] T 1ce1610623954041bc98ab3f4e5113f1 P ab3d1803f22146a99f38e3d02631abc2: Bootstrap starting.
I20260928 08:54:33.259262 2488 tablet_bootstrap.cc:492] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46: No bootstrap required, opened a new log
I20260928 08:54:33.259390 2488 ts_tablet_manager.cc:1397] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:54:33.259548 2489 tablet_bootstrap.cc:654] T 1ce1610623954041bc98ab3f4e5113f1 P ab3d1803f22146a99f38e3d02631abc2: Neither blocks nor log segments found. Creating new log.
I20260928 08:54:33.260059 2489 log.cc:824] T 1ce1610623954041bc98ab3f4e5113f1 P ab3d1803f22146a99f38e3d02631abc2: Log is configured to *not* fsync() on all Append() calls
I20260928 08:54:33.260820 2489 tablet_bootstrap.cc:492] T 1ce1610623954041bc98ab3f4e5113f1 P ab3d1803f22146a99f38e3d02631abc2: No bootstrap required, opened a new log
I20260928 08:54:33.260888 2489 ts_tablet_manager.cc:1397] T 1ce1610623954041bc98ab3f4e5113f1 P ab3d1803f22146a99f38e3d02631abc2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:54:33.261057 2487 raft_consensus.cc:348] T 1ce1610623954041bc98ab3f4e5113f1 P aec300b90f174e159db1453af76be440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec300b90f174e159db1453af76be440" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 37739 } } peers { permanent_uuid: "ab3d1803f22146a99f38e3d02631abc2" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34773 } } peers { permanent_uuid: "4f57de7b8dbd45d3b27577bcd4a1da46" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34103 } }
I20260928 08:54:33.261261 2487 raft_consensus.cc:374] T 1ce1610623954041bc98ab3f4e5113f1 P aec300b90f174e159db1453af76be440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:54:33.261170 2488 raft_consensus.cc:348] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec300b90f174e159db1453af76be440" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 37739 } } peers { permanent_uuid: "ab3d1803f22146a99f38e3d02631abc2" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34773 } } peers { permanent_uuid: "4f57de7b8dbd45d3b27577bcd4a1da46" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34103 } }
I20260928 08:54:33.261304 2487 raft_consensus.cc:729] T 1ce1610623954041bc98ab3f4e5113f1 P aec300b90f174e159db1453af76be440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec300b90f174e159db1453af76be440, State: Initialized, Role: FOLLOWER
I20260928 08:54:33.261317 2488 raft_consensus.cc:374] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:54:33.261349 2488 raft_consensus.cc:729] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f57de7b8dbd45d3b27577bcd4a1da46, State: Initialized, Role: FOLLOWER
I20260928 08:54:33.261416 2487 consensus_queue.cc:260] T 1ce1610623954041bc98ab3f4e5113f1 P aec300b90f174e159db1453af76be440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec300b90f174e159db1453af76be440" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 37739 } } peers { permanent_uuid: "ab3d1803f22146a99f38e3d02631abc2" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34773 } } peers { permanent_uuid: "4f57de7b8dbd45d3b27577bcd4a1da46" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34103 } }
I20260928 08:54:33.261459 2488 consensus_queue.cc:260] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec300b90f174e159db1453af76be440" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 37739 } } peers { permanent_uuid: "ab3d1803f22146a99f38e3d02631abc2" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34773 } } peers { permanent_uuid: "4f57de7b8dbd45d3b27577bcd4a1da46" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34103 } }
I20260928 08:54:33.261699 2488 ts_tablet_manager.cc:1428] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 08:54:33.261700 2487 ts_tablet_manager.cc:1428] T 1ce1610623954041bc98ab3f4e5113f1 P aec300b90f174e159db1453af76be440: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:54:33.261819 2473 heartbeater.cc:503] Master 127.23.167.254:38527 was elected leader, sending a full tablet report...
I20260928 08:54:33.261857 2205 heartbeater.cc:503] Master 127.23.167.254:38527 was elected leader, sending a full tablet report...
I20260928 08:54:33.263080 2489 raft_consensus.cc:348] T 1ce1610623954041bc98ab3f4e5113f1 P ab3d1803f22146a99f38e3d02631abc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec300b90f174e159db1453af76be440" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 37739 } } peers { permanent_uuid: "ab3d1803f22146a99f38e3d02631abc2" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34773 } } peers { permanent_uuid: "4f57de7b8dbd45d3b27577bcd4a1da46" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34103 } }
I20260928 08:54:33.263211 2489 raft_consensus.cc:374] T 1ce1610623954041bc98ab3f4e5113f1 P ab3d1803f22146a99f38e3d02631abc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:54:33.263231 2489 raft_consensus.cc:729] T 1ce1610623954041bc98ab3f4e5113f1 P ab3d1803f22146a99f38e3d02631abc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3d1803f22146a99f38e3d02631abc2, State: Initialized, Role: FOLLOWER
I20260928 08:54:33.263334 2489 consensus_queue.cc:260] T 1ce1610623954041bc98ab3f4e5113f1 P ab3d1803f22146a99f38e3d02631abc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec300b90f174e159db1453af76be440" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 37739 } } peers { permanent_uuid: "ab3d1803f22146a99f38e3d02631abc2" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34773 } } peers { permanent_uuid: "4f57de7b8dbd45d3b27577bcd4a1da46" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34103 } }
I20260928 08:54:33.263581 2489 ts_tablet_manager.cc:1428] T 1ce1610623954041bc98ab3f4e5113f1 P ab3d1803f22146a99f38e3d02631abc2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:54:33.263598 2339 heartbeater.cc:503] Master 127.23.167.254:38527 was elected leader, sending a full tablet report...
W20260928 08:54:33.316517 2340 tablet.cc:2369] T 1ce1610623954041bc98ab3f4e5113f1 P ab3d1803f22146a99f38e3d02631abc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:54:33.417614 2206 tablet.cc:2369] T 1ce1610623954041bc98ab3f4e5113f1 P aec300b90f174e159db1453af76be440: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:54:33.428349 2494 raft_consensus.cc:482] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:54:33.428508 2494 raft_consensus.cc:504] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec300b90f174e159db1453af76be440" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 37739 } } peers { permanent_uuid: "ab3d1803f22146a99f38e3d02631abc2" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34773 } } peers { permanent_uuid: "4f57de7b8dbd45d3b27577bcd4a1da46" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34103 } }
I20260928 08:54:33.428946 2494 leader_election.cc:288] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aec300b90f174e159db1453af76be440 (127.23.167.193:37739), ab3d1803f22146a99f38e3d02631abc2 (127.23.167.194:34773)
I20260928 08:54:33.432881 2475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:33.429179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:37739 (local address 127.23.167.195:39015)
0928 08:54:33.429238 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:33.429252 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:33.429303 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:33.429497 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:33.429498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:33.429500 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:54:33.429572 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:33.429574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:33.430489 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:33.430490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:33.431824 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:33.431828 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:33.432583 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:33.432585 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:33.432674 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:33.432676 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 08:54:33.432735 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 08:54:33.432790 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":46}
I20260928 08:54:33.432914 2486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:33.429355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:36253 (local address 127.23.167.194:34773)
0928 08:54:33.429394 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:33.429396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:33.429509 (+ 113us) server_negotiation.cc:407] Connection header received
0928 08:54:33.429541 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:33.429542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:33.429552 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:33.429592 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:54:33.429706 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:33.430391 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:33.431913 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:33.432504 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:33.432574 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:33.432584 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:33.432790 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 08:54:33.432837 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 08:54:33.432909 2483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:33.429232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:39015 (local address 127.23.167.193:37739)
0928 08:54:33.429300 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:33.429303 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:33.429313 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:54:33.429386 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:33.429388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:33.429400 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:33.429451 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:54:33.429641 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:33.430391 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:33.431912 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:33.432501 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:33.432574 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:33.432584 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:33.432792 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 08:54:33.432837 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 08:54:33.432888 2496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:33.429274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:34773 (local address 127.23.167.195:36253)
0928 08:54:33.429436 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:33.429445 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:54:33.429495 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:54:33.429594 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:54:33.429595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:54:33.429596 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:54:33.429651 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:33.429652 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:33.430474 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:33.430475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:33.431824 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:54:33.431827 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:54:33.432584 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:54:33.432584 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:54:33.432670 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:33.432675 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 08:54:33.432735 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 08:54:33.432790 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":63,"thread_start_us":112,"threads_started":1}
I20260928 08:54:33.433342 2159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ce1610623954041bc98ab3f4e5113f1" candidate_uuid: "4f57de7b8dbd45d3b27577bcd4a1da46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec300b90f174e159db1453af76be440" is_pre_election: true
I20260928 08:54:33.433506 2159 raft_consensus.cc:2454] T 1ce1610623954041bc98ab3f4e5113f1 P aec300b90f174e159db1453af76be440 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f57de7b8dbd45d3b27577bcd4a1da46 in term 0.
I20260928 08:54:33.433632 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ce1610623954041bc98ab3f4e5113f1" candidate_uuid: "4f57de7b8dbd45d3b27577bcd4a1da46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3d1803f22146a99f38e3d02631abc2" is_pre_election: true
I20260928 08:54:33.433779 2293 raft_consensus.cc:2454] T 1ce1610623954041bc98ab3f4e5113f1 P ab3d1803f22146a99f38e3d02631abc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f57de7b8dbd45d3b27577bcd4a1da46 in term 0.
I20260928 08:54:33.433756 2361 leader_election.cc:302] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f57de7b8dbd45d3b27577bcd4a1da46, aec300b90f174e159db1453af76be440; no voters:
I20260928 08:54:33.433871 2494 raft_consensus.cc:2781] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:54:33.433940 2494 raft_consensus.cc:482] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:54:33.433977 2494 raft_consensus.cc:3037] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:54:33.434689 2494 raft_consensus.cc:504] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec300b90f174e159db1453af76be440" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 37739 } } peers { permanent_uuid: "ab3d1803f22146a99f38e3d02631abc2" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34773 } } peers { permanent_uuid: "4f57de7b8dbd45d3b27577bcd4a1da46" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34103 } }
I20260928 08:54:33.434921 2494 leader_election.cc:288] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46 [CANDIDATE]: Term 1 election: Requested vote from peers aec300b90f174e159db1453af76be440 (127.23.167.193:37739), ab3d1803f22146a99f38e3d02631abc2 (127.23.167.194:34773)
I20260928 08:54:33.434959 2159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ce1610623954041bc98ab3f4e5113f1" candidate_uuid: "4f57de7b8dbd45d3b27577bcd4a1da46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec300b90f174e159db1453af76be440"
I20260928 08:54:33.435019 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ce1610623954041bc98ab3f4e5113f1" candidate_uuid: "4f57de7b8dbd45d3b27577bcd4a1da46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3d1803f22146a99f38e3d02631abc2"
I20260928 08:54:33.435045 2159 raft_consensus.cc:3037] T 1ce1610623954041bc98ab3f4e5113f1 P aec300b90f174e159db1453af76be440 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:54:33.435086 2293 raft_consensus.cc:3037] T 1ce1610623954041bc98ab3f4e5113f1 P ab3d1803f22146a99f38e3d02631abc2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:54:33.435912 2293 raft_consensus.cc:2454] T 1ce1610623954041bc98ab3f4e5113f1 P ab3d1803f22146a99f38e3d02631abc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f57de7b8dbd45d3b27577bcd4a1da46 in term 1.
I20260928 08:54:33.435912 2159 raft_consensus.cc:2454] T 1ce1610623954041bc98ab3f4e5113f1 P aec300b90f174e159db1453af76be440 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f57de7b8dbd45d3b27577bcd4a1da46 in term 1.
I20260928 08:54:33.436088 2361 leader_election.cc:302] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f57de7b8dbd45d3b27577bcd4a1da46, ab3d1803f22146a99f38e3d02631abc2; no voters:
I20260928 08:54:33.436189 2494 raft_consensus.cc:2781] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:54:33.436348 2494 raft_consensus.cc:686] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46 [term 1 LEADER]: Becoming Leader. State: Replica: 4f57de7b8dbd45d3b27577bcd4a1da46, State: Running, Role: LEADER
I20260928 08:54:33.436436 2494 consensus_queue.cc:237] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec300b90f174e159db1453af76be440" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 37739 } } peers { permanent_uuid: "ab3d1803f22146a99f38e3d02631abc2" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34773 } } peers { permanent_uuid: "4f57de7b8dbd45d3b27577bcd4a1da46" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34103 } }
I20260928 08:54:33.437327 2012 catalog_manager.cc:5495] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f57de7b8dbd45d3b27577bcd4a1da46 (127.23.167.195). New cstate: current_term: 1 leader_uuid: "4f57de7b8dbd45d3b27577bcd4a1da46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec300b90f174e159db1453af76be440" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 37739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3d1803f22146a99f38e3d02631abc2" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f57de7b8dbd45d3b27577bcd4a1da46" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34103 } health_report { overall_health: HEALTHY } } }
W20260928 08:54:33.469194 2474 tablet.cc:2369] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:54:33.478308 2484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:33.475984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45018 (local address 127.23.167.195:34103)
0928 08:54:33.476046 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 08:54:33.476048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:54:33.476057 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:54:33.476096 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:54:33.476096 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:54:33.476113 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:54:33.476174 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:54:33.476293 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:33.476771 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:33.477555 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:54:33.477749 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:54:33.477804 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:54:33.477962 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:54:33.478075 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:54:33.478148 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 08:54:33.478188 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 08:54:33.478208 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 08:54:33.479988 2293 raft_consensus.cc:1261] T 1ce1610623954041bc98ab3f4e5113f1 P ab3d1803f22146a99f38e3d02631abc2 [term 1 FOLLOWER]: Refusing update from remote peer 4f57de7b8dbd45d3b27577bcd4a1da46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:54:33.480005 2159 raft_consensus.cc:1261] T 1ce1610623954041bc98ab3f4e5113f1 P aec300b90f174e159db1453af76be440 [term 1 FOLLOWER]: Refusing update from remote peer 4f57de7b8dbd45d3b27577bcd4a1da46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:54:33.480353 2497 consensus_queue.cc:1035] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aec300b90f174e159db1453af76be440" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 37739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:54:33.480522 2494 consensus_queue.cc:1035] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3d1803f22146a99f38e3d02631abc2" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:54:33.482807 2500 mvcc.cc:205] Tried to move back new op lower bound from 7334238918571646976 to 7334238918396424192. Current Snapshot: MvccSnapshot[applied={T|T < 7334238918571646976}]
I20260928 08:54:33.483079 2502 mvcc.cc:205] Tried to move back new op lower bound from 7334238918571646976 to 7334238918396424192. Current Snapshot: MvccSnapshot[applied={T|T < 7334238918571646976}]
I20260928 08:54:33.485212 2012 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:54:33.485293 2012 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:54:33.486030 2012 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e633e5ffcd7497d89774f3b24b5f423: Sending DeleteTablet for 3 replicas of tablet 1ce1610623954041bc98ab3f4e5113f1
I20260928 08:54:33.486402 2407 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ce1610623954041bc98ab3f4e5113f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:54:33 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:45008
I20260928 08:54:33.486408 2273 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ce1610623954041bc98ab3f4e5113f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:54:33 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:50326
I20260928 08:54:33.486621 2139 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ce1610623954041bc98ab3f4e5113f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:54:33 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:39138
I20260928 08:54:33.486822 2512 tablet_replica.cc:330] T 1ce1610623954041bc98ab3f4e5113f1 P aec300b90f174e159db1453af76be440: stopping tablet replica
I20260928 08:54:33.486917 2510 tablet_replica.cc:330] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46: stopping tablet replica
I20260928 08:54:33.487016 2510 raft_consensus.cc:2229] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:54:33.487010 2512 raft_consensus.cc:2229] T 1ce1610623954041bc98ab3f4e5113f1 P aec300b90f174e159db1453af76be440 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:54:33.487112 2510 raft_consensus.cc:2258] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:54:33.487112 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 2076
I20260928 08:54:33.487248 2511 tablet_replica.cc:330] T 1ce1610623954041bc98ab3f4e5113f1 P ab3d1803f22146a99f38e3d02631abc2: stopping tablet replica
I20260928 08:54:33.487331 2511 raft_consensus.cc:2229] T 1ce1610623954041bc98ab3f4e5113f1 P ab3d1803f22146a99f38e3d02631abc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:54:33.487416 2511 raft_consensus.cc:2258] T 1ce1610623954041bc98ab3f4e5113f1 P ab3d1803f22146a99f38e3d02631abc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:54:33.487850 2511 ts_tablet_manager.cc:1905] T 1ce1610623954041bc98ab3f4e5113f1 P ab3d1803f22146a99f38e3d02631abc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:54:33.488227 2510 ts_tablet_manager.cc:1905] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:54:33.489677 2511 ts_tablet_manager.cc:1918] T 1ce1610623954041bc98ab3f4e5113f1 P ab3d1803f22146a99f38e3d02631abc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:54:33.489753 2511 log.cc:1196] T 1ce1610623954041bc98ab3f4e5113f1 P ab3d1803f22146a99f38e3d02631abc2: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-1/wal/wals/1ce1610623954041bc98ab3f4e5113f1
I20260928 08:54:33.490067 2511 ts_tablet_manager.cc:1939] T 1ce1610623954041bc98ab3f4e5113f1 P ab3d1803f22146a99f38e3d02631abc2: Deleting consensus metadata
I20260928 08:54:33.490293 2510 ts_tablet_manager.cc:1918] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:54:33.490427 2510 log.cc:1196] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585589999956-24223-0/minicluster-data/ts-2/wal/wals/1ce1610623954041bc98ab3f4e5113f1
I20260928 08:54:33.490598 1999 catalog_manager.cc:4841] TS ab3d1803f22146a99f38e3d02631abc2 (127.23.167.194:34773): tablet 1ce1610623954041bc98ab3f4e5113f1 (table test-table [id=df0e870abd924394abbd2b4f49f010cb]) successfully deleted
I20260928 08:54:33.490801 2510 ts_tablet_manager.cc:1939] T 1ce1610623954041bc98ab3f4e5113f1 P 4f57de7b8dbd45d3b27577bcd4a1da46: Deleting consensus metadata
I20260928 08:54:33.491279 2001 catalog_manager.cc:4841] TS 4f57de7b8dbd45d3b27577bcd4a1da46 (127.23.167.195:34103): tablet 1ce1610623954041bc98ab3f4e5113f1 (table test-table [id=df0e870abd924394abbd2b4f49f010cb]) successfully deleted
W20260928 08:54:33.492823 1999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.167.193:37739 (error 108)
I20260928 08:54:33.494076 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 2210
I20260928 08:54:33.497565 2482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:54:33.497407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:37739 (local address 127.0.0.1:39146)
0928 08:54:33.497474 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 08:54:33.497526 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.193:37739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 08:54:33.497711 1999 catalog_manager.cc:4584] TS aec300b90f174e159db1453af76be440 (127.23.167.193:37739): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1ce1610623954041bc98ab3f4e5113f1: Network error: Client connection negotiation failed: client connection to 127.23.167.193:37739: connect: Connection refused (error 111)
I20260928 08:54:33.499020 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 2344
I20260928 08:54:33.503826 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 1982
2026-09-28T08:54:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1961 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:54:33 dist-test-slave-pbjp krb5kdc[2516](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:54:33 dist-test-slave-pbjp krb5kdc[2516](info): set up 2 sockets
Sep 28 08:54:33 dist-test-slave-pbjp krb5kdc[2516](info): commencing operation
krb5kdc: starting...
W20260928 08:54:34.452898 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:54:34 dist-test-slave-pbjp krb5kdc[2516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:54:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:54:34Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.167.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.167.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.167.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/rangeradmin_127.23.167.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.167.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/rangeradmin_127.23.167.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.167.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.167.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.167.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/rangerlookup_127.23.167.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.167.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/rangerlookup_127.23.167.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.167.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.167.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.167.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/HTTP_127.23.167.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.167.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/HTTP_127.23.167.212@KRBTEST.COM.keytab.
I20260928 08:54:34.604161 24223 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskN7pK8W/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/postgres -l logfile start
2026-09-28 08:54:36.964 UTC [2546] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 08:54:36.964 UTC [2546] LOG: listening on IPv4 address "127.23.167.212", port 34529
2026-09-28 08:54:36.967 UTC [2546] LOG: listening on Unix socket "/tmp/.s.PGSQL.34529"
2026-09-28 08:54:36.987 UTC [2549] LOG: database system was shut down at 2026-09-28 08:54:35 UTC
2026-09-28 08:54:36.990 UTC [2546] LOG: database system is ready to accept connections
I20260928 08:54:37.860877 24223 mini_postgres.cc:96] Postgres bound to 34529
2026-09-28 08:54:37.866 UTC [2557] FATAL: database "slave" does not exist
127.23.167.212:34529 - accepting connections
I20260928 08:54:37.867393 24223 mini_ranger.cc:161] Starting Ranger...
I20260928 08:54:37.881367 24223 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 08:54:38.317929 24223 mini_ranger.cc:87] Created ranger Postgres database
I20260928 08:54:38.317992 24223 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ranger-admin
2026-09-28 08:54:38,623 [I] DB FLAVOR :POSTGRES
2026-09-28 08:54:38,623 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 08:54:38,624 [I] Checking connection..
2026-09-28 08:54:38,624 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:34529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 08:54:38,885 [I] Checking connection passed.
2026-09-28 08:54:38,885 [I] --------- Verifying version history table ---------
2026-09-28 08:54:38,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:34529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 08:54:39,116 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 08:54:39,116 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:34529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 08:54:39,344 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 08:54:39,344 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 08:54:39,344 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:34529/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 08:54:39.592 UTC [2621] WARNING: there is no transaction in progress
2026-09-28 08:54:39,597 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 08:54:39,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:34529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 08:54:39,832 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 08:54:39,832 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 08:54:39,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:34529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 08:54:40,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:34529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 08:54:40,347 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:34529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-pbjp.c.gcp-upstream.internal','N') ;"
2026-09-28 08:54:40,577 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:34529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 08:54:40,805 [I] Table x_portal_user does not exist in database ranger
2026-09-28 08:54:40,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:34529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 08:54:41,052 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 08:54:41,053 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:34529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 08:54:41,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:34529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 08:54:41,553 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 08:54:41,553 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:34529/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 08:54:42.498 UTC [2729] WARNING: there is no transaction in progress
2026-09-28 08:54:42.513 UTC [2729] WARNING: there is no transaction in progress
2026-09-28 08:54:42.529 UTC [2729] WARNING: there is no transaction in progress
2026-09-28 08:54:42.544 UTC [2729] WARNING: there is no transaction in progress
2026-09-28 08:54:42.560 UTC [2729] WARNING: there is no transaction in progress
2026-09-28 08:54:42.578 UTC [2729] WARNING: there is no transaction in progress
2026-09-28 08:54:42.642 UTC [2729] WARNING: there is no transaction in progress
2026-09-28 08:54:42.649 UTC [2729] WARNING: there is no transaction in progress
2026-09-28 08:54:42.663 UTC [2729] WARNING: there is no transaction in progress
2026-09-28 08:54:42.673 UTC [2729] WARNING: there is no transaction in progress
2026-09-28 08:54:42.683 UTC [2729] WARNING: there is no transaction in progress
2026-09-28 08:54:42.696 UTC [2729] WARNING: there is no transaction in progress
2026-09-28 08:54:42.705 UTC [2729] WARNING: there is no transaction in progress
2026-09-28 08:54:42.711 UTC [2729] WARNING: there is no transaction in progress
2026-09-28 08:54:42.718 UTC [2729] WARNING: there is no transaction in progress
2026-09-28 08:54:42.724 UTC [2729] WARNING: there is no transaction in progress
2026-09-28 08:54:43,329 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 08:54:43,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:34529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 08:54:43,572 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 08:54:43,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:34529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 08:54:43,807 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 08:54:43,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:34529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 08:54:44,047 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 08:54:44,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:34529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 08:54:44,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:34529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 08:54:44,548 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:34529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-pbjp.c.gcp-upstream.internal';"
2026-09-28 08:54:44,778 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 08:54:44,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:34529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 08:54:45,018 [I] DB_PATCHES have already been applied
I20260928 08:54:45.023504 24223 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ranger-admin:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskN7pK8W/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 08:54:45.023568 24223 mini_ranger.cc:193] Using host: 127.23.167.212
I20260928 08:54:45.026424 24223 mini_ranger.cc:227] Ranger admin URL: http://127.23.167.212:51415
Sep 28, 2026 8:54:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 8:54:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 8:54:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 8:54:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 8:54:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 8:54:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 8:54:45 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 08:54:46.011125 24223 mini_ranger.cc:160] Time spent starting Ranger: real 8.144s user 0.001s sys 0.007s
Sep 28, 2026 8:55:05 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 8:55:05 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 28, 2026 8:55:05 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 8:55:05 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 8:55:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 8:55:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 8:55:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 8:55:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 8:55:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 8:55:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 8:55:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 8:55:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 8:55:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 8:55:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 8:55:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 8:55:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 8:55:05 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 8:55:06 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 08:55:07.253100 24223 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.167.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.167.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.167.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/rangerkms_127.23.167.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.167.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/rangerkms_127.23.167.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.167.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.167.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.167.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/HTTP_127.23.167.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.167.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/HTTP_127.23.167.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 08:55:07 dist-test-slave-pbjp krb5kdc[2516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585707, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 08:55:07.332644 24223 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 08:55:07.342200 24223 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 08:55:08.110733 24223 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 08:55:08.110816 24223 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ranger-kms
I20260928 08:55:08.110829 24223 mini_ranger_kms.cc:227] Using postgres at 127.23.167.212:34529
2026-09-28 08:55:08,293 [I] DB FLAVOR :POSTGRES
2026-09-28 08:55:08,293 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 08:55:08,293 [I] Checking connection
2026-09-28 08:55:08,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:34529/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 08:55:08,529 [I] connection success
2026-09-28 08:55:08,530 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 08:55:08,530 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 08:55:08,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:34529/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 08:55:08,757 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 08:55:08,757 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 08:55:08,757 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 08:55:08,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:34529/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 08:55:09,004 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 08:55:09.449363 24223 mini_ranger_kms.cc:310] Created kms service
I20260928 08:55:09.639993 24223 mini_ranger_kms.cc:326] Created kudu user
I20260928 08:55:09.684525 24223 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 08:55:09 dist-test-slave-pbjp krb5kdc[2516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585707, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.167.212@KRBTEST.COM
I20260928 08:55:10.069391 24223 mini_ranger_kms.cc:384] Added ranger policy
I20260928 08:55:10.069679 24223 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ranger-kms:/tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 08:55:10.069703 24223 mini_ranger_kms.cc:240] Using host: 127.23.167.212
I20260928 08:55:10.074096 24223 mini_ranger_kms.cc:276] Ranger KMS PID: 2909
I20260928 08:55:10.074213 24223 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.167.212:53189
Sep 28, 2026 8:55:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 8:55:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 8:55:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 8:55:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 8:55:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 8:55:11 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 08:55:12.096309 24223 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.764s user 0.009s sys 0.156s
I20260928 08:55:12.096407 24223 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 08:55:12.096480 24223 mini_ranger_kms.cc:401] 127.23.167.212:53189/kms/v1/keys
Sep 28 08:55:12 dist-test-slave-pbjp krb5kdc[2516](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790585712, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.167.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 08:55:12 dist-test-slave-pbjp krb5kdc[2516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585712, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.167.212@KRBTEST.COM for HTTP/127.23.167.212@KRBTEST.COM
Sep 28 08:55:12 dist-test-slave-pbjp krb5kdc[2516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585712, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.167.212@KRBTEST.COM for HTTP/127.23.167.212@KRBTEST.COM
Sep 28 08:55:15 dist-test-slave-pbjp krb5kdc[2516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.167.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
I20260928 08:55:15.228309 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:37729
--webserver_interface=127.23.167.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:37951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:37729
--ranger_config_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.167.212:53189/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:55:15.311961 2960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:55:15.312160 2960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:55:15.312187 2960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:55:15.313427 2960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:55:15.313468 2960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:55:15.313481 2960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:55:15.313493 2960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:55:15.313504 2960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:55:15.315505 2960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:37951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.167.212:53189/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:37729
--ranger_config_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:37729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:55:15.315745 2960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:55:15.315955 2960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:55:15.320880 2968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:55:15.320963 2960 server_base.cc:1034] running on GCE node
W20260928 08:55:15.320894 2966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:55:15.320881 2965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:55:15.321367 2960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:55:15.321659 2960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:55:15.322794 2960 hybrid_clock.cc:648] HybridClock initialized: now 1790585715322779 us; error 27 us; skew 500 ppm
Sep 28 08:55:15 dist-test-slave-pbjp krb5kdc[2516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585715, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:55:15.330544 2960 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:55:15.331002 2960 webserver.cc:460] Webserver started at http://127.23.167.254:46507/ using document root <none> and password file <none>
I20260928 08:55:15.331251 2960 fs_manager.cc:359] Metadata directory not provided
I20260928 08:55:15.331293 2960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:55:15.331398 2960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 08:55:15 dist-test-slave-pbjp krb5kdc[2516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585715, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for HTTP/127.23.167.212@KRBTEST.COM
I20260928 08:55:15.385993 2960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "cbf1d85fd7a946aba9f30d860ab03d1b"
format_stamp: "Formatted at 2026-09-28 08:55:15 on dist-test-slave-pbjp"
server_key: "be1c402be390a6441bd4ac27d3237eff"
server_key_iv: "898f681611a98c6ed370fc73fcf9e547"
server_key_version: "kuduclusterkey@0"
I20260928 08:55:15.386371 2960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "cbf1d85fd7a946aba9f30d860ab03d1b"
format_stamp: "Formatted at 2026-09-28 08:55:15 on dist-test-slave-pbjp"
server_key: "be1c402be390a6441bd4ac27d3237eff"
server_key_iv: "898f681611a98c6ed370fc73fcf9e547"
server_key_version: "kuduclusterkey@0"
I20260928 08:55:15.387683 2960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:55:15.402146 2975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:55:15.402407 2960 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:55:15.402526 2960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "cbf1d85fd7a946aba9f30d860ab03d1b"
format_stamp: "Formatted at 2026-09-28 08:55:15 on dist-test-slave-pbjp"
server_key: "be1c402be390a6441bd4ac27d3237eff"
server_key_iv: "898f681611a98c6ed370fc73fcf9e547"
server_key_version: "kuduclusterkey@0"
I20260928 08:55:15.402629 2960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:55:15.415608 2960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:55:15.417593 2960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:55:15.417742 2960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:55:15.422370 2960 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:37729
I20260928 08:55:15.422381 3027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:37729 every 8 connection(s)
I20260928 08:55:15.422786 2960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:55:15.423470 3028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:55:15.423990 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 2960
I20260928 08:55:15.424086 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:55:15.424278 24223 external_mini_cluster.cc:1428] Setting key 94366a01c9ba8c6e31fe860df90954d5
I20260928 08:55:15.426134 3028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf1d85fd7a946aba9f30d860ab03d1b: Bootstrap starting.
I20260928 08:55:15.426671 3028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbf1d85fd7a946aba9f30d860ab03d1b: Neither blocks nor log segments found. Creating new log.
I20260928 08:55:15.426957 3028 log.cc:824] T 00000000000000000000000000000000 P cbf1d85fd7a946aba9f30d860ab03d1b: Log is configured to *not* fsync() on all Append() calls
I20260928 08:55:15.427703 3028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf1d85fd7a946aba9f30d860ab03d1b: No bootstrap required, opened a new log
I20260928 08:55:15.430234 3028 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbf1d85fd7a946aba9f30d860ab03d1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf1d85fd7a946aba9f30d860ab03d1b" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 37729 } }
I20260928 08:55:15.430413 3028 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbf1d85fd7a946aba9f30d860ab03d1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:55:15.430529 3028 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbf1d85fd7a946aba9f30d860ab03d1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbf1d85fd7a946aba9f30d860ab03d1b, State: Initialized, Role: FOLLOWER
I20260928 08:55:15.430711 3028 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbf1d85fd7a946aba9f30d860ab03d1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf1d85fd7a946aba9f30d860ab03d1b" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 37729 } }
I20260928 08:55:15.430826 3028 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbf1d85fd7a946aba9f30d860ab03d1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:55:15.430876 3028 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbf1d85fd7a946aba9f30d860ab03d1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:55:15.430950 3028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbf1d85fd7a946aba9f30d860ab03d1b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:55:15.431656 3028 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbf1d85fd7a946aba9f30d860ab03d1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf1d85fd7a946aba9f30d860ab03d1b" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 37729 } }
I20260928 08:55:15.431767 3028 leader_election.cc:302] T 00000000000000000000000000000000 P cbf1d85fd7a946aba9f30d860ab03d1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbf1d85fd7a946aba9f30d860ab03d1b; no voters:
I20260928 08:55:15.431936 3028 leader_election.cc:288] T 00000000000000000000000000000000 P cbf1d85fd7a946aba9f30d860ab03d1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:55:15.432058 3033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbf1d85fd7a946aba9f30d860ab03d1b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:55:15.432273 3028 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbf1d85fd7a946aba9f30d860ab03d1b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:55:15.432284 3033 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbf1d85fd7a946aba9f30d860ab03d1b [term 1 LEADER]: Becoming Leader. State: Replica: cbf1d85fd7a946aba9f30d860ab03d1b, State: Running, Role: LEADER
I20260928 08:55:15.432498 3033 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbf1d85fd7a946aba9f30d860ab03d1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf1d85fd7a946aba9f30d860ab03d1b" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 37729 } }
Sep 28 08:55:15 dist-test-slave-pbjp krb5kdc[2516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:55:15.432821 3033 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf1d85fd7a946aba9f30d860ab03d1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbf1d85fd7a946aba9f30d860ab03d1b. Latest consensus state: current_term: 1 leader_uuid: "cbf1d85fd7a946aba9f30d860ab03d1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf1d85fd7a946aba9f30d860ab03d1b" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 37729 } } }
I20260928 08:55:15.433048 3033 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf1d85fd7a946aba9f30d860ab03d1b [sys.catalog]: This master's current role is: LEADER
I20260928 08:55:15.432988 3035 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf1d85fd7a946aba9f30d860ab03d1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbf1d85fd7a946aba9f30d860ab03d1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf1d85fd7a946aba9f30d860ab03d1b" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 37729 } } }
I20260928 08:55:15.433314 3035 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf1d85fd7a946aba9f30d860ab03d1b [sys.catalog]: This master's current role is: LEADER
I20260928 08:55:15.434111 3042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:55:15.433898 3028 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 08:55:15.434623 3042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:55:15.436954 3042 catalog_manager.cc:1347] Generated new cluster ID: 784904c23e00436db6a21f3951e2a236
I20260928 08:55:15.437040 3042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:55:15.437346 3031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:55:15.424833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43778 (local address 127.23.167.254:37729)
0928 08:55:15.427198 (+ 2365us) server_negotiation.cc:206] Beginning negotiation
0928 08:55:15.427207 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:55:15.427241 (+ 34us) server_negotiation.cc:407] Connection header received
0928 08:55:15.427656 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:55:15.427664 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:55:15.427715 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:55:15.427857 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:55:15.428477 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:55:15.429396 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:55:15.430129 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:55:15.430615 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:55:15.430718 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:55:15.433434 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:55:15.433438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:55:15.433438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:55:15.433452 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:55:15.435567 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:55:15.436015 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:55:15.436019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:55:15.436019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:55:15.436117 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:55:15.436329 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:55:15.436330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:55:15.436330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:55:15.436645 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:55:15.436790 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 08:55:15.436973 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 08:55:15.437127 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2318,"thread_start_us":71,"threads_started":1}
I20260928 08:55:15.450094 3042 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:55:15.450930 3042 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:55:15.464133 3042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbf1d85fd7a946aba9f30d860ab03d1b: Generated new TSK 0
I20260928 08:55:15.464457 3042 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.167.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
I20260928 08:55:16.152364 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.167.212:53189/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:0
--local_ip_for_outbound_sockets=127.23.167.193
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:37729
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:37951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:55:16.270761 3073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:55:16.271032 3073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:55:16.271070 3073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:55:16.272662 3073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:55:16.272722 3073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:55:16.272758 3073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:55:16.274858 3073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:37951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.167.212:53189/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:37729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:55:16.275112 3073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:55:16.275403 3073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:55:16.280668 3079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:55:16.280787 3073 server_base.cc:1034] running on GCE node
W20260928 08:55:16.280812 3081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:55:16.280659 3078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:55:16.281116 3073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:55:16.281361 3073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:55:16.282519 3073 hybrid_clock.cc:648] HybridClock initialized: now 1790585716282505 us; error 29 us; skew 500 ppm
Sep 28 08:55:16 dist-test-slave-pbjp krb5kdc[2516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585716, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:55:16.284952 3073 init.cc:377] Logged in from keytab as kudu/127.23.167.193@KRBTEST.COM (short username kudu)
I20260928 08:55:16.285403 3073 webserver.cc:460] Webserver started at http://127.23.167.193:34905/ using document root <none> and password file <none>
I20260928 08:55:16.285601 3073 fs_manager.cc:359] Metadata directory not provided
I20260928 08:55:16.285650 3073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:55:16.285766 3073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 08:55:16 dist-test-slave-pbjp krb5kdc[2516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585716, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for HTTP/127.23.167.212@KRBTEST.COM
I20260928 08:55:16.301362 3073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-0/data/instance:
uuid: "5c3ee6af19a543719e4426e6adf756a5"
format_stamp: "Formatted at 2026-09-28 08:55:16 on dist-test-slave-pbjp"
server_key: "781d8094d4b184728fcab35dd8d42be6"
server_key_iv: "9c6c581cbe0a40fcf5ed4b646120316a"
server_key_version: "kuduclusterkey@0"
I20260928 08:55:16.301796 3073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance:
uuid: "5c3ee6af19a543719e4426e6adf756a5"
format_stamp: "Formatted at 2026-09-28 08:55:16 on dist-test-slave-pbjp"
server_key: "781d8094d4b184728fcab35dd8d42be6"
server_key_iv: "9c6c581cbe0a40fcf5ed4b646120316a"
server_key_version: "kuduclusterkey@0"
I20260928 08:55:16.304037 3073 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 08:55:16.323357 3088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:55:16.323616 3073 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.000s
I20260928 08:55:16.323726 3073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "5c3ee6af19a543719e4426e6adf756a5"
format_stamp: "Formatted at 2026-09-28 08:55:16 on dist-test-slave-pbjp"
server_key: "781d8094d4b184728fcab35dd8d42be6"
server_key_iv: "9c6c581cbe0a40fcf5ed4b646120316a"
server_key_version: "kuduclusterkey@0"
I20260928 08:55:16.323848 3073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:55:16.354097 3073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:55:16.356519 3073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:55:16.356732 3073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:55:16.357021 3073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:55:16.357560 3073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:55:16.357649 3073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:55:16.357748 3073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:55:16.357817 3073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:55:16.364593 3073 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:41915
I20260928 08:55:16.364748 3203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:41915 every 8 connection(s)
I20260928 08:55:16.365303 3073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
I20260928 08:55:16.370225 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 3073
I20260928 08:55:16.370324 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance
I20260928 08:55:16.370509 24223 external_mini_cluster.cc:1428] Setting key 5237aabefe9bae58a5e09977f2fe01cc
Sep 28 08:55:16 dist-test-slave-pbjp krb5kdc[2516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585716, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:55:16.378396 3207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:55:16.366114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:37729 (local address 127.23.167.193:57303)
0928 08:55:16.366430 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 08:55:16.366476 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 08:55:16.367117 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:55:16.367458 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:55:16.367462 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:55:16.367691 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:55:16.367986 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:55:16.367993 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:55:16.369022 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:55:16.369023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:55:16.370365 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:55:16.370368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:55:16.371057 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:55:16.371058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:55:16.371207 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:55:16.371790 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:55:16.371800 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:55:16.373403 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:55:16.376704 (+ 3301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:55:16.376712 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:55:16.376715 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:55:16.377013 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:55:16.377272 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:55:16.377273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:55:16.377273 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:55:16.377406 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:55:16.377715 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:55:16.377722 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:55:16.377896 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 08:55:16.377994 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 08:55:16.378151 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for kudu/127.23.167.194@KRBTEST.COM; defaulting to no policy
I20260928 08:55:16.379180 3206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:55:16.366123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:57303 (local address 127.23.167.254:37729)
0928 08:55:16.366332 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 08:55:16.366334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:55:16.366583 (+ 249us) server_negotiation.cc:407] Connection header received
0928 08:55:16.367247 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:55:16.367249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:55:16.367265 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:55:16.367355 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:55:16.368100 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:55:16.368917 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:55:16.370465 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:55:16.370921 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:55:16.370985 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:55:16.373505 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:55:16.373507 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:55:16.373507 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:55:16.373515 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:55:16.376600 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:55:16.377127 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:55:16.377128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:55:16.377128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:55:16.377197 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:55:16.377493 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:55:16.377494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:55:16.377494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:55:16.377636 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:55:16.378940 (+ 1304us) server_negotiation.cc:1035] Waiting for connection context
0928 08:55:16.378987 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 08:55:16.379072 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":81,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.194@KRBTEST.COM" created.
I20260928 08:55:16.385226 3204 heartbeater.cc:347] Connected to a master server at 127.23.167.254:37729
I20260928 08:55:16.385325 3204 heartbeater.cc:464] Registering TS with master...
I20260928 08:55:16.385524 3204 heartbeater.cc:511] Master 127.23.167.254:37729 requested a full tablet report, sending...
I20260928 08:55:16.386162 2992 ts_manager.cc:194] Registered new tserver with Master: 5c3ee6af19a543719e4426e6adf756a5 (127.23.167.193:41915)
I20260928 08:55:16.387046 2992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.193@KRBTEST.COM'} at 127.23.167.193:57303
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
I20260928 08:55:16.403097 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.167.212:53189/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:0
--local_ip_for_outbound_sockets=127.23.167.194
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:37729
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:37951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:55:16.521202 3210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:55:16.521584 3210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:55:16.521687 3210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:55:16.523430 3210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:55:16.523658 3210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:55:16.523757 3210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:55:16.525992 3210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:37951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.167.212:53189/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:37729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:55:16.526440 3210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:55:16.526765 3210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:55:16.533849 3220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:55:16.533852 3218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:55:16.534489 3217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:55:16.534701 3210 server_base.cc:1034] running on GCE node
I20260928 08:55:16.535084 3210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:55:16.535370 3210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:55:16.536556 3210 hybrid_clock.cc:648] HybridClock initialized: now 1790585716536528 us; error 38 us; skew 500 ppm
Sep 28 08:55:16 dist-test-slave-pbjp krb5kdc[2516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585716, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:55:16.539987 3210 init.cc:377] Logged in from keytab as kudu/127.23.167.194@KRBTEST.COM (short username kudu)
I20260928 08:55:16.540526 3210 webserver.cc:460] Webserver started at http://127.23.167.194:35729/ using document root <none> and password file <none>
I20260928 08:55:16.540771 3210 fs_manager.cc:359] Metadata directory not provided
I20260928 08:55:16.540823 3210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:55:16.540944 3210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 08:55:16 dist-test-slave-pbjp krb5kdc[2516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585716, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for HTTP/127.23.167.212@KRBTEST.COM
I20260928 08:55:16.557216 3210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-1/data/instance:
uuid: "0786493cfe794ba8bf266c6672d1469c"
format_stamp: "Formatted at 2026-09-28 08:55:16 on dist-test-slave-pbjp"
server_key: "714623a54540185e5679213ceb273d2a"
server_key_iv: "d28e5b9d63709c5863519806d6b046e5"
server_key_version: "kuduclusterkey@0"
I20260928 08:55:16.557675 3210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance:
uuid: "0786493cfe794ba8bf266c6672d1469c"
format_stamp: "Formatted at 2026-09-28 08:55:16 on dist-test-slave-pbjp"
server_key: "714623a54540185e5679213ceb273d2a"
server_key_iv: "d28e5b9d63709c5863519806d6b046e5"
server_key_version: "kuduclusterkey@0"
I20260928 08:55:16.559494 3210 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
Sep 28 08:55:16 dist-test-slave-pbjp krb5kdc[2516](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790585716, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:55:16.573966 3227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:55:16.574229 3210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 08:55:16.574323 3210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "0786493cfe794ba8bf266c6672d1469c"
format_stamp: "Formatted at 2026-09-28 08:55:16 on dist-test-slave-pbjp"
server_key: "714623a54540185e5679213ceb273d2a"
server_key_iv: "d28e5b9d63709c5863519806d6b046e5"
server_key_version: "kuduclusterkey@0"
I20260928 08:55:16.574419 3210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:55:16.595305 3210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:55:16.597738 3210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:55:16.597932 3210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:55:16.598214 3210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:55:16.598691 3210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:55:16.598776 3210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:55:16.598847 3210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:55:16.598904 3210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:55:16.607407 3210 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:40427
I20260928 08:55:16.607935 3210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
I20260928 08:55:16.608651 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 3210
I20260928 08:55:16.608741 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance
I20260928 08:55:16.608911 24223 external_mini_cluster.cc:1428] Setting key 5b6c098f6f6a32747c530b16c10d1700
I20260928 08:55:16.613574 3340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:40427 every 8 connection(s)
Sep 28 08:55:16 dist-test-slave-pbjp krb5kdc[2516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585716, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.167.195@KRBTEST.COM; defaulting to no policy
I20260928 08:55:16.626048 3206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:55:16.613369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:44991 (local address 127.23.167.254:37729)
0928 08:55:16.613415 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 08:55:16.613417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:55:16.615768 (+ 2351us) server_negotiation.cc:407] Connection header received
0928 08:55:16.616693 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:55:16.616695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:55:16.616713 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:55:16.616743 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:55:16.617551 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:55:16.618047 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:55:16.619561 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:55:16.619881 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:55:16.619926 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:55:16.622608 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:55:16.622610 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:55:16.622610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:55:16.622618 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:55:16.624322 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:55:16.624922 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:55:16.624923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:55:16.624923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:55:16.624979 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:55:16.625288 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:55:16.625289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:55:16.625289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:55:16.625431 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:55:16.625483 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 08:55:16.625844 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 08:55:16.625937 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.195@KRBTEST.COM" created.
I20260928 08:55:16.626803 3344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:55:16.613042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:37729 (local address 127.23.167.194:44991)
0928 08:55:16.615620 (+ 2578us) negotiation.cc:107] Waiting for socket to connect
0928 08:55:16.615672 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 08:55:16.616474 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:55:16.616939 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:55:16.616944 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:55:16.617227 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:55:16.617440 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:55:16.617445 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:55:16.618141 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:55:16.618142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:55:16.619453 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:55:16.619456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:55:16.620036 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:55:16.620037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:55:16.620186 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:55:16.620781 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:55:16.620792 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:55:16.622500 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:55:16.624552 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:55:16.624559 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:55:16.624561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:55:16.624838 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:55:16.625055 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:55:16.625056 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:55:16.625056 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:55:16.625208 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:55:16.625516 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:55:16.625524 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:55:16.625689 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 08:55:16.626393 (+ 704us) client_negotiation.cc:240] Negotiation successful
0928 08:55:16.626562 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2516,"thread_start_us":92,"threads_started":1}
I20260928 08:55:16.627467 3341 heartbeater.cc:347] Connected to a master server at 127.23.167.254:37729
I20260928 08:55:16.627566 3341 heartbeater.cc:464] Registering TS with master...
I20260928 08:55:16.627759 3341 heartbeater.cc:511] Master 127.23.167.254:37729 requested a full tablet report, sending...
I20260928 08:55:16.628252 2992 ts_manager.cc:194] Registered new tserver with Master: 0786493cfe794ba8bf266c6672d1469c (127.23.167.194:40427)
I20260928 08:55:16.628736 2992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.194@KRBTEST.COM'} at 127.23.167.194:44991
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
I20260928 08:55:16.652422 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.167.212:53189/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:0
--local_ip_for_outbound_sockets=127.23.167.195
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:37729
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:37951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:55:16.776188 3346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:55:16.776512 3346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:55:16.776557 3346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:55:16.778419 3346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:55:16.778479 3346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:55:16.778519 3346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:55:16.780881 3346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:37951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.167.212:53189/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:37729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:55:16.781167 3346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:55:16.781459 3346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:55:16.788002 3355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:55:16.788009 3353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:55:16.788338 3352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:55:16.788760 3346 server_base.cc:1034] running on GCE node
I20260928 08:55:16.788985 3346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:55:16.789312 3346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:55:16.790510 3346 hybrid_clock.cc:648] HybridClock initialized: now 1790585716790480 us; error 43 us; skew 500 ppm
Sep 28 08:55:16 dist-test-slave-pbjp krb5kdc[2516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585716, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:55:16.794132 3346 init.cc:377] Logged in from keytab as kudu/127.23.167.195@KRBTEST.COM (short username kudu)
I20260928 08:55:16.794795 3346 webserver.cc:460] Webserver started at http://127.23.167.195:41243/ using document root <none> and password file <none>
I20260928 08:55:16.795061 3346 fs_manager.cc:359] Metadata directory not provided
I20260928 08:55:16.795136 3346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:55:16.795276 3346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 08:55:16 dist-test-slave-pbjp krb5kdc[2516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585716, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for HTTP/127.23.167.212@KRBTEST.COM
I20260928 08:55:16.810249 3346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-2/data/instance:
uuid: "07e507a2fbc74502bb0ffbf37817b355"
format_stamp: "Formatted at 2026-09-28 08:55:16 on dist-test-slave-pbjp"
server_key: "902674cf49a5769d283245304367ba9b"
server_key_iv: "5cf5d1297634e5613f8490500db7c06b"
server_key_version: "kuduclusterkey@0"
I20260928 08:55:16.811033 3346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance:
uuid: "07e507a2fbc74502bb0ffbf37817b355"
format_stamp: "Formatted at 2026-09-28 08:55:16 on dist-test-slave-pbjp"
server_key: "902674cf49a5769d283245304367ba9b"
server_key_iv: "5cf5d1297634e5613f8490500db7c06b"
server_key_version: "kuduclusterkey@0"
I20260928 08:55:16.814267 3346 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:55:16.833415 3362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:55:16.833698 3346 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260928 08:55:16.833811 3346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "07e507a2fbc74502bb0ffbf37817b355"
format_stamp: "Formatted at 2026-09-28 08:55:16 on dist-test-slave-pbjp"
server_key: "902674cf49a5769d283245304367ba9b"
server_key_iv: "5cf5d1297634e5613f8490500db7c06b"
server_key_version: "kuduclusterkey@0"
I20260928 08:55:16.833923 3346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:55:16.846437 3346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:55:16.848786 3346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:55:16.848948 3346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:55:16.849223 3346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:55:16.849599 3346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:55:16.849658 3346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:55:16.849689 3346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:55:16.849754 3346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:55:16.859036 3346 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:44009
I20260928 08:55:16.859622 3346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
I20260928 08:55:16.863914 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 3346
I20260928 08:55:16.864012 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance
I20260928 08:55:16.864187 24223 external_mini_cluster.cc:1428] Setting key ba0c5ee5638f5cb702186f1a694d90b1
I20260928 08:55:16.861323 3475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:44009 every 8 connection(s)
Sep 28 08:55:16 dist-test-slave-pbjp krb5kdc[2516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585716, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:55:16.873399 3206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:55:16.862347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:47143 (local address 127.23.167.254:37729)
0928 08:55:16.862388 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 08:55:16.862390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:55:16.862686 (+ 296us) server_negotiation.cc:407] Connection header received
0928 08:55:16.863422 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:55:16.863423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:55:16.863435 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:55:16.863465 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:55:16.864501 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:55:16.865300 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:55:16.867027 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:55:16.867334 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:55:16.867377 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:55:16.870078 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:55:16.870080 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:55:16.870080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:55:16.870088 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:55:16.871691 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:55:16.872185 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:55:16.872187 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:55:16.872187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:55:16.872247 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:55:16.872527 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:55:16.872528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:55:16.872529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:55:16.872668 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:55:16.872717 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 08:55:16.873218 (+ 501us) server_negotiation.cc:299] Negotiation successful
0928 08:55:16.873302 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 08:55:16.874176 3478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:55:16.862059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:37729 (local address 127.23.167.195:47143)
0928 08:55:16.862538 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 08:55:16.862596 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 08:55:16.863292 (+ 696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:55:16.863664 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:55:16.863669 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:55:16.863915 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:55:16.864398 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:55:16.864406 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:55:16.865591 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:55:16.865591 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:55:16.866899 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:55:16.866903 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:55:16.867468 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:55:16.867469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:55:16.867614 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:55:16.868131 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:55:16.868142 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:55:16.869958 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:55:16.871790 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:55:16.871796 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:55:16.871799 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:55:16.872090 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:55:16.872313 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:55:16.872314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:55:16.872314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:55:16.872451 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:55:16.872765 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:55:16.872771 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:55:16.873020 (+ 249us) client_negotiation.cc:769] Sending connection context
0928 08:55:16.873747 (+ 727us) client_negotiation.cc:240] Negotiation successful
0928 08:55:16.873916 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":82,"threads_started":1}
I20260928 08:55:16.874817 3476 heartbeater.cc:347] Connected to a master server at 127.23.167.254:37729
I20260928 08:55:16.874909 3476 heartbeater.cc:464] Registering TS with master...
I20260928 08:55:16.875106 3476 heartbeater.cc:511] Master 127.23.167.254:37729 requested a full tablet report, sending...
I20260928 08:55:16.875538 2992 ts_manager.cc:194] Registered new tserver with Master: 07e507a2fbc74502bb0ffbf37817b355 (127.23.167.195:44009)
I20260928 08:55:16.876014 2992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.195@KRBTEST.COM'} at 127.23.167.195:47143
I20260928 08:55:16.876031 24223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:55:16.886883 3206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:55:16.877739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43788 (local address 127.23.167.254:37729)
0928 08:55:16.877774 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 08:55:16.877776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:55:16.877884 (+ 108us) server_negotiation.cc:407] Connection header received
0928 08:55:16.877971 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:55:16.877971 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:55:16.877994 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:55:16.878021 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:55:16.878737 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:55:16.879497 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:55:16.880563 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:55:16.880860 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:55:16.880922 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:55:16.882507 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:55:16.882508 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:55:16.882509 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:55:16.882516 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:55:16.885761 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:55:16.886122 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:55:16.886123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:55:16.886123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:55:16.886182 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:55:16.886348 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:55:16.886349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:55:16.886350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:55:16.886492 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:55:16.886583 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 08:55:16.886714 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 08:55:16.886789 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 08:55:16.888460 2992 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:55:16.890190 2992 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:55:16.897461 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:55:16.892572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:40427 (local address 127.0.0.1:39070)
0928 08:55:16.893269 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0928 08:55:16.893278 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:55:16.893310 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:55:16.893626 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:55:16.893629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:55:16.893631 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:55:16.893698 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:55:16.893699 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:55:16.894940 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:55:16.894941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:55:16.896213 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:55:16.896215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:55:16.897183 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:55:16.897184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:55:16.897263 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:55:16.897279 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 08:55:16.897324 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 08:55:16.897375 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":91,"threads_started":1}
I20260928 08:55:16.897929 3491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:55:16.892940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39070 (local address 127.23.167.194:40427)
0928 08:55:16.893476 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0928 08:55:16.893480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:55:16.893489 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:55:16.893520 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:55:16.893522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:55:16.893533 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:55:16.893598 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:55:16.893776 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:55:16.894830 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:55:16.896328 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:55:16.897095 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:55:16.897705 (+ 610us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:55:16.897772 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 08:55:16.897799 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 08:55:16.897838 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":77,"threads_started":1}
I20260928 08:55:16.898123 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:55:16.892571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:41915 (local address 127.0.0.1:47238)
0928 08:55:16.893031 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 08:55:16.893098 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 08:55:16.893163 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:55:16.893871 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:55:16.893872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:55:16.893874 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:55:16.893910 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:55:16.893911 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:55:16.895464 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:55:16.895465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:55:16.896507 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:55:16.896509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:55:16.897923 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:55:16.897924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:55:16.897987 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:55:16.897991 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:55:16.898021 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 08:55:16.898058 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"spinlock_wait_cycles":1280,"thread_start_us":93,"threads_started":1}
I20260928 08:55:16.898670 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:55:16.893134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47238 (local address 127.23.167.193:41915)
0928 08:55:16.893720 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0928 08:55:16.893724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:55:16.893733 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:55:16.893764 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:55:16.893767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:55:16.893781 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:55:16.893975 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:55:16.894032 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:55:16.895358 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:55:16.896837 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:55:16.897838 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:55:16.898468 (+ 630us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:55:16.898533 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 08:55:16.898559 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 08:55:16.898602 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":70,"threads_started":1}
I20260928 08:55:16.898617 3275 tablet_service.cc:1467] Processing CreateTablet for tablet cf4564f7c1d6494fb28fc0e1cbfcc7da (DEFAULT_TABLE table=test-table [id=76f518e4732645ad9b80fe35bb1143d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:55:16.899021 3275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf4564f7c1d6494fb28fc0e1cbfcc7da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:55:16.901842 3138 tablet_service.cc:1467] Processing CreateTablet for tablet cf4564f7c1d6494fb28fc0e1cbfcc7da (DEFAULT_TABLE table=test-table [id=76f518e4732645ad9b80fe35bb1143d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:55:16.902218 3138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf4564f7c1d6494fb28fc0e1cbfcc7da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:55:16.904627 3494 tablet_bootstrap.cc:492] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 5c3ee6af19a543719e4426e6adf756a5: Bootstrap starting.
I20260928 08:55:16.905557 3495 tablet_bootstrap.cc:492] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 0786493cfe794ba8bf266c6672d1469c: Bootstrap starting.
I20260928 08:55:16.905882 3494 tablet_bootstrap.cc:654] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 5c3ee6af19a543719e4426e6adf756a5: Neither blocks nor log segments found. Creating new log.
I20260928 08:55:16.906306 3494 log.cc:824] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 5c3ee6af19a543719e4426e6adf756a5: Log is configured to *not* fsync() on all Append() calls
I20260928 08:55:16.906649 3495 tablet_bootstrap.cc:654] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 0786493cfe794ba8bf266c6672d1469c: Neither blocks nor log segments found. Creating new log.
I20260928 08:55:16.906965 3495 log.cc:824] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 0786493cfe794ba8bf266c6672d1469c: Log is configured to *not* fsync() on all Append() calls
I20260928 08:55:16.907246 3494 tablet_bootstrap.cc:492] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 5c3ee6af19a543719e4426e6adf756a5: No bootstrap required, opened a new log
I20260928 08:55:16.907310 3494 ts_tablet_manager.cc:1397] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 5c3ee6af19a543719e4426e6adf756a5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 08:55:16.907718 3495 tablet_bootstrap.cc:492] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 0786493cfe794ba8bf266c6672d1469c: No bootstrap required, opened a new log
I20260928 08:55:16.907775 3495 ts_tablet_manager.cc:1397] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 0786493cfe794ba8bf266c6672d1469c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:55:16.909322 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:55:16.892894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:44009 (local address 127.0.0.1:53276)
0928 08:55:16.893359 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 08:55:16.893364 (+ 5us) client_negotiation.cc:174] Beginning negotiation
0928 08:55:16.893387 (+ 23us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:55:16.905387 (+ 12000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:55:16.905389 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:55:16.905392 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:55:16.905453 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:55:16.905455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:55:16.906845 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:55:16.906847 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:55:16.908033 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:55:16.908037 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:55:16.909045 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:55:16.909047 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:55:16.909155 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:55:16.909161 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 08:55:16.909190 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 08:55:16.909240 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":66,"threads_started":1}
I20260928 08:55:16.909760 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:55:16.893026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53276 (local address 127.23.167.195:44009)
0928 08:55:16.905184 (+ 12158us) server_negotiation.cc:206] Beginning negotiation
0928 08:55:16.905189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:55:16.905203 (+ 14us) server_negotiation.cc:407] Connection header received
0928 08:55:16.905243 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:55:16.905245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:55:16.905263 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:55:16.905522 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:55:16.905599 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:55:16.906726 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:55:16.908171 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:55:16.908933 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:55:16.909536 (+ 603us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:55:16.909611 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 08:55:16.909637 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 08:55:16.909675 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12109,"thread_start_us":123,"threads_started":1}
I20260928 08:55:16.910010 3495 raft_consensus.cc:348] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 0786493cfe794ba8bf266c6672d1469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3ee6af19a543719e4426e6adf756a5" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 41915 } } peers { permanent_uuid: "07e507a2fbc74502bb0ffbf37817b355" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44009 } } peers { permanent_uuid: "0786493cfe794ba8bf266c6672d1469c" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 40427 } }
I20260928 08:55:16.910183 3495 raft_consensus.cc:374] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 0786493cfe794ba8bf266c6672d1469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:55:16.910100 3494 raft_consensus.cc:348] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 5c3ee6af19a543719e4426e6adf756a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3ee6af19a543719e4426e6adf756a5" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 41915 } } peers { permanent_uuid: "07e507a2fbc74502bb0ffbf37817b355" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44009 } } peers { permanent_uuid: "0786493cfe794ba8bf266c6672d1469c" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 40427 } }
I20260928 08:55:16.910216 3495 raft_consensus.cc:729] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 0786493cfe794ba8bf266c6672d1469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0786493cfe794ba8bf266c6672d1469c, State: Initialized, Role: FOLLOWER
I20260928 08:55:16.910221 3494 raft_consensus.cc:374] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 5c3ee6af19a543719e4426e6adf756a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:55:16.910249 3494 raft_consensus.cc:729] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 5c3ee6af19a543719e4426e6adf756a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c3ee6af19a543719e4426e6adf756a5, State: Initialized, Role: FOLLOWER
I20260928 08:55:16.910338 3495 consensus_queue.cc:260] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 0786493cfe794ba8bf266c6672d1469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3ee6af19a543719e4426e6adf756a5" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 41915 } } peers { permanent_uuid: "07e507a2fbc74502bb0ffbf37817b355" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44009 } } peers { permanent_uuid: "0786493cfe794ba8bf266c6672d1469c" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 40427 } }
I20260928 08:55:16.910406 3409 tablet_service.cc:1467] Processing CreateTablet for tablet cf4564f7c1d6494fb28fc0e1cbfcc7da (DEFAULT_TABLE table=test-table [id=76f518e4732645ad9b80fe35bb1143d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:55:16.910616 3495 ts_tablet_manager.cc:1428] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 0786493cfe794ba8bf266c6672d1469c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:55:16.910741 3341 heartbeater.cc:503] Master 127.23.167.254:37729 was elected leader, sending a full tablet report...
I20260928 08:55:16.910738 3409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf4564f7c1d6494fb28fc0e1cbfcc7da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:55:16.911237 3494 consensus_queue.cc:260] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 5c3ee6af19a543719e4426e6adf756a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3ee6af19a543719e4426e6adf756a5" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 41915 } } peers { permanent_uuid: "07e507a2fbc74502bb0ffbf37817b355" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44009 } } peers { permanent_uuid: "0786493cfe794ba8bf266c6672d1469c" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 40427 } }
I20260928 08:55:16.911615 3494 ts_tablet_manager.cc:1428] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 5c3ee6af19a543719e4426e6adf756a5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 08:55:16.911768 3204 heartbeater.cc:503] Master 127.23.167.254:37729 was elected leader, sending a full tablet report...
I20260928 08:55:16.914084 3500 tablet_bootstrap.cc:492] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355: Bootstrap starting.
I20260928 08:55:16.915011 3500 tablet_bootstrap.cc:654] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355: Neither blocks nor log segments found. Creating new log.
I20260928 08:55:16.915316 3500 log.cc:824] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355: Log is configured to *not* fsync() on all Append() calls
I20260928 08:55:16.916170 3500 tablet_bootstrap.cc:492] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355: No bootstrap required, opened a new log
I20260928 08:55:16.916232 3500 ts_tablet_manager.cc:1397] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:55:16.918201 3500 raft_consensus.cc:348] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3ee6af19a543719e4426e6adf756a5" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 41915 } } peers { permanent_uuid: "07e507a2fbc74502bb0ffbf37817b355" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44009 } } peers { permanent_uuid: "0786493cfe794ba8bf266c6672d1469c" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 40427 } }
I20260928 08:55:16.918382 3500 raft_consensus.cc:374] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:55:16.918411 3500 raft_consensus.cc:729] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07e507a2fbc74502bb0ffbf37817b355, State: Initialized, Role: FOLLOWER
I20260928 08:55:16.918556 3500 consensus_queue.cc:260] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3ee6af19a543719e4426e6adf756a5" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 41915 } } peers { permanent_uuid: "07e507a2fbc74502bb0ffbf37817b355" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44009 } } peers { permanent_uuid: "0786493cfe794ba8bf266c6672d1469c" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 40427 } }
I20260928 08:55:16.918815 3500 ts_tablet_manager.cc:1428] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:55:16.918969 3476 heartbeater.cc:503] Master 127.23.167.254:37729 was elected leader, sending a full tablet report...
Sep 28 08:55:17 dist-test-slave-pbjp krb5kdc[2516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585716, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for HTTP/127.23.167.212@KRBTEST.COM
I20260928 08:55:17.098230 3502 raft_consensus.cc:482] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:55:17.098380 3502 raft_consensus.cc:504] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3ee6af19a543719e4426e6adf756a5" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 41915 } } peers { permanent_uuid: "07e507a2fbc74502bb0ffbf37817b355" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44009 } } peers { permanent_uuid: "0786493cfe794ba8bf266c6672d1469c" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 40427 } }
I20260928 08:55:17.098846 3502 leader_election.cc:288] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c3ee6af19a543719e4426e6adf756a5 (127.23.167.193:41915), 0786493cfe794ba8bf266c6672d1469c (127.23.167.194:40427)
I20260928 08:55:17.103368 3478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:55:17.099212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:41915 (local address 127.23.167.195:37469)
0928 08:55:17.099317 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 08:55:17.099328 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:55:17.099387 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:55:17.100058 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:55:17.100059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:55:17.100073 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:55:17.100217 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:55:17.100221 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:55:17.101177 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:55:17.101178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:55:17.102347 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:55:17.102351 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:55:17.103130 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:55:17.103130 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:55:17.103212 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:55:17.103216 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:55:17.103263 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 08:55:17.103298 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 08:55:17.103730 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:55:17.099510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:37469 (local address 127.23.167.193:41915)
0928 08:55:17.099558 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 08:55:17.099560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:55:17.099568 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:55:17.099595 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:55:17.099596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:55:17.099605 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:55:17.099653 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:55:17.100307 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:55:17.101083 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:55:17.102455 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:55:17.103020 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:55:17.103090 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:55:17.103099 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 08:55:17.103621 (+ 522us) server_negotiation.cc:299] Negotiation successful
0928 08:55:17.103665 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 08:55:17.103993 3158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf4564f7c1d6494fb28fc0e1cbfcc7da" candidate_uuid: "07e507a2fbc74502bb0ffbf37817b355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c3ee6af19a543719e4426e6adf756a5" is_pre_election: true
I20260928 08:55:17.104147 3158 raft_consensus.cc:2454] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 5c3ee6af19a543719e4426e6adf756a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07e507a2fbc74502bb0ffbf37817b355 in term 0.
I20260928 08:55:17.104389 3363 leader_election.cc:302] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07e507a2fbc74502bb0ffbf37817b355, 5c3ee6af19a543719e4426e6adf756a5; no voters:
I20260928 08:55:17.104542 3502 raft_consensus.cc:2781] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:55:17.104589 3502 raft_consensus.cc:482] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:55:17.104612 3502 raft_consensus.cc:3037] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:55:17.105573 3502 raft_consensus.cc:504] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3ee6af19a543719e4426e6adf756a5" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 41915 } } peers { permanent_uuid: "07e507a2fbc74502bb0ffbf37817b355" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44009 } } peers { permanent_uuid: "0786493cfe794ba8bf266c6672d1469c" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 40427 } }
I20260928 08:55:17.105710 3502 leader_election.cc:288] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355 [CANDIDATE]: Term 1 election: Requested vote from peers 5c3ee6af19a543719e4426e6adf756a5 (127.23.167.193:41915), 0786493cfe794ba8bf266c6672d1469c (127.23.167.194:40427)
I20260928 08:55:17.105918 3158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf4564f7c1d6494fb28fc0e1cbfcc7da" candidate_uuid: "07e507a2fbc74502bb0ffbf37817b355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c3ee6af19a543719e4426e6adf756a5"
I20260928 08:55:17.105993 3158 raft_consensus.cc:3037] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 5c3ee6af19a543719e4426e6adf756a5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:55:17.106894 3158 raft_consensus.cc:2454] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 5c3ee6af19a543719e4426e6adf756a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07e507a2fbc74502bb0ffbf37817b355 in term 1.
I20260928 08:55:17.107169 3363 leader_election.cc:302] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07e507a2fbc74502bb0ffbf37817b355, 5c3ee6af19a543719e4426e6adf756a5; no voters:
I20260928 08:55:17.107273 3502 raft_consensus.cc:2781] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:55:17.107417 3502 raft_consensus.cc:686] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355 [term 1 LEADER]: Becoming Leader. State: Replica: 07e507a2fbc74502bb0ffbf37817b355, State: Running, Role: LEADER
I20260928 08:55:17.107513 3502 consensus_queue.cc:237] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3ee6af19a543719e4426e6adf756a5" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 41915 } } peers { permanent_uuid: "07e507a2fbc74502bb0ffbf37817b355" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44009 } } peers { permanent_uuid: "0786493cfe794ba8bf266c6672d1469c" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 40427 } }
I20260928 08:55:17.108414 2992 catalog_manager.cc:5495] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07e507a2fbc74502bb0ffbf37817b355 (127.23.167.195). New cstate: current_term: 1 leader_uuid: "07e507a2fbc74502bb0ffbf37817b355" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3ee6af19a543719e4426e6adf756a5" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 41915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07e507a2fbc74502bb0ffbf37817b355" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 44009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0786493cfe794ba8bf266c6672d1469c" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 40427 } health_report { overall_health: UNKNOWN } } }
W20260928 08:55:17.111682 3477 tablet.cc:2369] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:55:17.112479 3478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:55:17.099799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:40427 (local address 127.23.167.195:54691)
0928 08:55:17.103457 (+ 3658us) negotiation.cc:107] Waiting for socket to connect
0928 08:55:17.103468 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:55:17.103510 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:55:17.108917 (+ 5407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:55:17.108919 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:55:17.108921 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:55:17.108962 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:55:17.108964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:55:17.110413 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:55:17.110414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:55:17.111513 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:55:17.111517 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:55:17.112259 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:55:17.112260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:55:17.112313 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:55:17.112317 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:55:17.112362 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 08:55:17.112404 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3646,"thread_start_us":105,"threads_started":1}
I20260928 08:55:17.112751 3491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:55:17.100003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:54691 (local address 127.23.167.194:40427)
0928 08:55:17.100049 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 08:55:17.100051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:55:17.103632 (+ 3581us) server_negotiation.cc:407] Connection header received
0928 08:55:17.103661 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:55:17.103662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:55:17.103675 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:55:17.103721 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:55:17.109122 (+ 5401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:55:17.109827 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:55:17.111618 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:55:17.112144 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:55:17.112217 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:55:17.112228 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 08:55:17.112643 (+ 415us) server_negotiation.cc:299] Negotiation successful
0928 08:55:17.112688 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 08:55:17.113091 3292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf4564f7c1d6494fb28fc0e1cbfcc7da" candidate_uuid: "07e507a2fbc74502bb0ffbf37817b355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0786493cfe794ba8bf266c6672d1469c" is_pre_election: true
I20260928 08:55:17.113255 3292 raft_consensus.cc:2454] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 0786493cfe794ba8bf266c6672d1469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07e507a2fbc74502bb0ffbf37817b355 in term 0.
I20260928 08:55:17.113377 3293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf4564f7c1d6494fb28fc0e1cbfcc7da" candidate_uuid: "07e507a2fbc74502bb0ffbf37817b355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0786493cfe794ba8bf266c6672d1469c"
I20260928 08:55:17.113438 3293 raft_consensus.cc:3037] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 0786493cfe794ba8bf266c6672d1469c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:55:17.114370 3293 raft_consensus.cc:2454] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 0786493cfe794ba8bf266c6672d1469c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07e507a2fbc74502bb0ffbf37817b355 in term 1.
W20260928 08:55:17.114758 3342 tablet.cc:2369] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 0786493cfe794ba8bf266c6672d1469c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:55:17.117195 3205 tablet.cc:2369] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 5c3ee6af19a543719e4426e6adf756a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:55:17.129230 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:55:17.125753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53286 (local address 127.23.167.195:44009)
0928 08:55:17.125800 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 08:55:17.125802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:55:17.125812 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:55:17.125894 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:55:17.125895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:55:17.125928 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:55:17.126005 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:55:17.126199 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:55:17.126921 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:55:17.128059 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:55:17.128388 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:55:17.128444 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:55:17.128682 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:55:17.128818 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:55:17.128896 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 08:55:17.129109 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 08:55:17.129130 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 08:55:17.132818 3293 raft_consensus.cc:1261] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 0786493cfe794ba8bf266c6672d1469c [term 1 FOLLOWER]: Refusing update from remote peer 07e507a2fbc74502bb0ffbf37817b355: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:55:17.133260 3504 consensus_queue.cc:1035] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0786493cfe794ba8bf266c6672d1469c" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 40427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:55:17.135802 3158 raft_consensus.cc:1261] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 5c3ee6af19a543719e4426e6adf756a5 [term 1 FOLLOWER]: Refusing update from remote peer 07e507a2fbc74502bb0ffbf37817b355: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:55:17.136093 3504 consensus_queue.cc:1035] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c3ee6af19a543719e4426e6adf756a5" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:55:17.140381 3507 mvcc.cc:205] Tried to move back new op lower bound from 7334239097372753920 to 7334239097272922112. Current Snapshot: MvccSnapshot[applied={T|T < 7334239097372753920}]
I20260928 08:55:17.141108 3510 mvcc.cc:205] Tried to move back new op lower bound from 7334239097372753920 to 7334239097272922112. Current Snapshot: MvccSnapshot[applied={T|T < 7334239097372753920}]
I20260928 08:55:17.143817 2992 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:55:17.143921 2992 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:55:17.145500 2992 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cbf1d85fd7a946aba9f30d860ab03d1b: Sending DeleteTablet for 3 replicas of tablet cf4564f7c1d6494fb28fc0e1cbfcc7da
I20260928 08:55:17.146217 3138 tablet_service.cc:1514] Processing DeleteTablet for tablet cf4564f7c1d6494fb28fc0e1cbfcc7da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:55:17 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:47238
I20260928 08:55:17.146548 3275 tablet_service.cc:1514] Processing DeleteTablet for tablet cf4564f7c1d6494fb28fc0e1cbfcc7da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:55:17 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:39070
I20260928 08:55:17.146592 3516 tablet_replica.cc:330] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 5c3ee6af19a543719e4426e6adf756a5: stopping tablet replica
I20260928 08:55:17.146831 3516 raft_consensus.cc:2229] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 5c3ee6af19a543719e4426e6adf756a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:55:17.147060 3516 raft_consensus.cc:2258] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 5c3ee6af19a543719e4426e6adf756a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:55:17.147228 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 3073
I20260928 08:55:17.149148 3409 tablet_service.cc:1514] Processing DeleteTablet for tablet cf4564f7c1d6494fb28fc0e1cbfcc7da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:55:17 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:53276
I20260928 08:55:17.149519 3518 tablet_replica.cc:330] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355: stopping tablet replica
I20260928 08:55:17.149662 3518 raft_consensus.cc:2229] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:55:17.149816 3518 raft_consensus.cc:2258] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:55:17.150447 3518 ts_tablet_manager.cc:1905] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:55:17.152937 3518 ts_tablet_manager.cc:1918] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:55:17.153121 3518 log.cc:1196] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585589999956-24223-0/minicluster-data/ts-2/wal/wals/cf4564f7c1d6494fb28fc0e1cbfcc7da
I20260928 08:55:17.153609 3518 ts_tablet_manager.cc:1939] T cf4564f7c1d6494fb28fc0e1cbfcc7da P 07e507a2fbc74502bb0ffbf37817b355: Deleting consensus metadata
I20260928 08:55:17.154199 2978 catalog_manager.cc:4841] TS 07e507a2fbc74502bb0ffbf37817b355 (127.23.167.195:44009): tablet cf4564f7c1d6494fb28fc0e1cbfcc7da (table test-table [id=76f518e4732645ad9b80fe35bb1143d9]) successfully deleted
W20260928 08:55:17.157498 2976 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.167.193:41915 (error 108)
I20260928 08:55:17.157859 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 3210
I20260928 08:55:17.169560 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:55:17.169408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:41915 (local address 127.0.0.1:47246)
0928 08:55:17.169465 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 08:55:17.169521 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.193:41915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 08:55:17.169725 2976 catalog_manager.cc:4584] TS 5c3ee6af19a543719e4426e6adf756a5 (127.23.167.193:41915): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cf4564f7c1d6494fb28fc0e1cbfcc7da: Network error: Client connection negotiation failed: client connection to 127.23.167.193:41915: connect: Connection refused (error 111)
I20260928 08:55:17.177028 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:55:17.176920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:40427 (local address 127.0.0.1:39082)
0928 08:55:17.176965 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 08:55:17.176987 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.194:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 08:55:17.177238 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 3346
I20260928 08:55:17.183248 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 2960
I20260928 08:55:17.188040 24223 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 08:55:17.522542 24223 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 08:55:17.522652 24223 mini_ranger.cc:66] Stopping Ranger...
I20260928 08:55:17.856570 24223 mini_ranger.cc:68] Stopped Ranger
2026-09-28 08:55:17.856 UTC [2546] LOG: received smart shutdown request
2026-09-28 08:55:17.859 UTC [2546] LOG: background worker "logical replication launcher" (PID 2555) exited with exit code 1
2026-09-28 08:55:17.860 UTC [2550] LOG: shutting down
2026-09-28 08:55:18.004 UTC [2546] LOG: database system is shut down
2026-09-28T08:55:18Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44586 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:55:18 dist-test-slave-pbjp krb5kdc[3528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:55:18 dist-test-slave-pbjp krb5kdc[3528](info): set up 2 sockets
Sep 28 08:55:18 dist-test-slave-pbjp krb5kdc[3528](info): commencing operation
krb5kdc: starting...
W20260928 08:55:19.036832 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:55:19 dist-test-slave-pbjp krb5kdc[3528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:55:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:55:19Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.167.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.167.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.167.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/rangeradmin_127.23.167.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.167.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/rangeradmin_127.23.167.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.167.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.167.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.167.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/rangerlookup_127.23.167.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.167.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/rangerlookup_127.23.167.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.167.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.167.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.167.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/HTTP_127.23.167.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.167.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/HTTP_127.23.167.212@KRBTEST.COM.keytab.
I20260928 08:55:19.199144 24223 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskN7pK8W/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/postgres -l logfile start
2026-09-28 08:55:21.521 UTC [3558] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 08:55:21.521 UTC [3558] LOG: listening on IPv4 address "127.23.167.212", port 57575
2026-09-28 08:55:21.524 UTC [3558] LOG: listening on Unix socket "/tmp/.s.PGSQL.57575"
2026-09-28 08:55:21.551 UTC [3561] LOG: database system was shut down at 2026-09-28 08:55:20 UTC
2026-09-28 08:55:21.554 UTC [3558] LOG: database system is ready to accept connections
I20260928 08:55:22.424645 24223 mini_postgres.cc:96] Postgres bound to 57575
2026-09-28 08:55:22.430 UTC [3569] FATAL: database "slave" does not exist
127.23.167.212:57575 - accepting connections
I20260928 08:55:22.431005 24223 mini_ranger.cc:161] Starting Ranger...
I20260928 08:55:22.445012 24223 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 08:55:22.879319 24223 mini_ranger.cc:87] Created ranger Postgres database
I20260928 08:55:22.879400 24223 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ranger-admin
2026-09-28 08:55:23,151 [I] DB FLAVOR :POSTGRES
2026-09-28 08:55:23,151 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 08:55:23,151 [I] Checking connection..
2026-09-28 08:55:23,151 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:57575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 08:55:23,397 [I] Checking connection passed.
2026-09-28 08:55:23,398 [I] --------- Verifying version history table ---------
2026-09-28 08:55:23,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:57575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 08:55:23,656 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 08:55:23,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:57575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 08:55:23,913 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 08:55:23,913 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 08:55:23,913 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:57575/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 08:55:24.143 UTC [3633] WARNING: there is no transaction in progress
2026-09-28 08:55:24,151 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 08:55:24,151 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:57575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 08:55:24,409 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 08:55:24,409 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 08:55:24,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:57575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 08:55:24,659 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:57575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 08:55:24,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:57575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-pbjp.c.gcp-upstream.internal','N') ;"
2026-09-28 08:55:25,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:57575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 08:55:25,396 [I] Table x_portal_user does not exist in database ranger
2026-09-28 08:55:25,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:57575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 08:55:25,655 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 08:55:25,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:57575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 08:55:25,891 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:57575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 08:55:26,146 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 08:55:26,146 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:57575/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 08:55:27.086 UTC [3741] WARNING: there is no transaction in progress
2026-09-28 08:55:27.102 UTC [3741] WARNING: there is no transaction in progress
2026-09-28 08:55:27.117 UTC [3741] WARNING: there is no transaction in progress
2026-09-28 08:55:27.132 UTC [3741] WARNING: there is no transaction in progress
2026-09-28 08:55:27.147 UTC [3741] WARNING: there is no transaction in progress
2026-09-28 08:55:27.164 UTC [3741] WARNING: there is no transaction in progress
2026-09-28 08:55:27.218 UTC [3741] WARNING: there is no transaction in progress
2026-09-28 08:55:27.224 UTC [3741] WARNING: there is no transaction in progress
2026-09-28 08:55:27.238 UTC [3741] WARNING: there is no transaction in progress
2026-09-28 08:55:27.248 UTC [3741] WARNING: there is no transaction in progress
2026-09-28 08:55:27.259 UTC [3741] WARNING: there is no transaction in progress
2026-09-28 08:55:27.272 UTC [3741] WARNING: there is no transaction in progress
2026-09-28 08:55:27.282 UTC [3741] WARNING: there is no transaction in progress
2026-09-28 08:55:27.289 UTC [3741] WARNING: there is no transaction in progress
2026-09-28 08:55:27.295 UTC [3741] WARNING: there is no transaction in progress
2026-09-28 08:55:27.302 UTC [3741] WARNING: there is no transaction in progress
2026-09-28 08:55:27,906 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 08:55:27,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:57575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 08:55:28,155 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 08:55:28,155 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:57575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 08:55:28,414 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 08:55:28,415 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:57575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 08:55:28,649 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 08:55:28,649 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:57575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 08:55:28,903 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:57575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 08:55:29,133 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:57575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-pbjp.c.gcp-upstream.internal';"
2026-09-28 08:55:29,356 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 08:55:29,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:57575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 08:55:29,579 [I] DB_PATCHES have already been applied
I20260928 08:55:29.584715 24223 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ranger-admin:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskN7pK8W/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 08:55:29.584779 24223 mini_ranger.cc:193] Using host: 127.23.167.212
I20260928 08:55:29.587576 24223 mini_ranger.cc:227] Ranger admin URL: http://127.23.167.212:55129
Sep 28, 2026 8:55:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 8:55:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 8:55:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 8:55:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 8:55:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 8:55:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 8:55:30 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 08:55:30.574705 24223 mini_ranger.cc:160] Time spent starting Ranger: real 8.144s user 0.003s sys 0.004s
Sep 28, 2026 8:55:50 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 8:55:50 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 8:55:50 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 8:55:50 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 8:55:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 8:55:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 8:55:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 8:55:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 8:55:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 8:55:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 8:55:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 8:55:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 8:55:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 8:55:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 8:55:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 8:55:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 8:55:50 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 8:55:51 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 08:55:52.032847 24223 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.167.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.167.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.167.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/rangerkms_127.23.167.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.167.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/rangerkms_127.23.167.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.167.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.167.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.167.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/HTTP_127.23.167.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.167.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/HTTP_127.23.167.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 08:55:52 dist-test-slave-pbjp krb5kdc[3528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585752, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 08:55:52.114368 24223 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 08:55:52.123215 24223 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 08:55:52.980851 24223 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 08:55:52.980917 24223 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ranger-kms
I20260928 08:55:52.980928 24223 mini_ranger_kms.cc:227] Using postgres at 127.23.167.212:57575
2026-09-28 08:55:53,125 [I] DB FLAVOR :POSTGRES
2026-09-28 08:55:53,125 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 08:55:53,125 [I] Checking connection
2026-09-28 08:55:53,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:57575/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 08:55:53,361 [I] connection success
2026-09-28 08:55:53,361 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 08:55:53,361 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 08:55:53,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:57575/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 08:55:53,606 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 08:55:53,606 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 08:55:53,607 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 08:55:53,607 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.167.212:57575/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 08:55:53,853 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 08:55:54.303828 24223 mini_ranger_kms.cc:310] Created kms service
I20260928 08:55:54.501698 24223 mini_ranger_kms.cc:326] Created kudu user
I20260928 08:55:54.543745 24223 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 08:55:54 dist-test-slave-pbjp krb5kdc[3528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585752, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.167.212@KRBTEST.COM
I20260928 08:55:54.944940 24223 mini_ranger_kms.cc:384] Added ranger policy
I20260928 08:55:54.945052 24223 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ranger-kms:/tmp/dist-test-taskN7pK8W/build/release/bin/postgresql.jar:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskN7pK8W/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 08:55:54.945096 24223 mini_ranger_kms.cc:240] Using host: 127.23.167.212
I20260928 08:55:54.948318 24223 mini_ranger_kms.cc:276] Ranger KMS PID: 3921
I20260928 08:55:54.948392 24223 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.167.212:46241
Sep 28, 2026 8:55:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 8:55:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 8:55:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 8:55:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 8:55:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 8:55:56 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 08:55:57.012055 24223 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.898s user 0.000s sys 0.127s
I20260928 08:55:57.012137 24223 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 08:55:57.012182 24223 mini_ranger_kms.cc:401] 127.23.167.212:46241/kms/v1/keys
Sep 28 08:55:57 dist-test-slave-pbjp krb5kdc[3528](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790585757, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.167.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 08:55:57 dist-test-slave-pbjp krb5kdc[3528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585757, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.167.212@KRBTEST.COM for HTTP/127.23.167.212@KRBTEST.COM
Sep 28 08:55:57 dist-test-slave-pbjp krb5kdc[3528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585757, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.167.212@KRBTEST.COM for HTTP/127.23.167.212@KRBTEST.COM
Sep 28 08:56:00 dist-test-slave-pbjp krb5kdc[3528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.167.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
I20260928 08:56:00.119088 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:40389
--webserver_interface=127.23.167.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:39765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:40389
--ranger_config_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.167.212:46241/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:56:00.199168 3972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:56:00.199369 3972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:56:00.199406 3972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:56:00.200555 3972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:56:00.200596 3972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:56:00.200608 3972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:56:00.200619 3972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:56:00.200632 3972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:56:00.202190 3972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:39765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.167.212:46241/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:40389
--ranger_config_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:40389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:56:00.202415 3972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:56:00.202626 3972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 08:56:00.208771 3972 server_base.cc:1034] running on GCE node
W20260928 08:56:00.208786 3980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:00.208737 3978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:00.208724 3977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:00.209394 3972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:56:00.209722 3972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:56:00.210870 3972 hybrid_clock.cc:648] HybridClock initialized: now 1790585760210855 us; error 32 us; skew 500 ppm
Sep 28 08:56:00 dist-test-slave-pbjp krb5kdc[3528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585760, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:00.218743 3972 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:56:00.219229 3972 webserver.cc:460] Webserver started at http://127.23.167.254:33125/ using document root <none> and password file <none>
I20260928 08:56:00.219460 3972 fs_manager.cc:359] Metadata directory not provided
I20260928 08:56:00.219501 3972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:56:00.219615 3972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 08:56:00 dist-test-slave-pbjp krb5kdc[3528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585760, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for HTTP/127.23.167.212@KRBTEST.COM
I20260928 08:56:00.274246 3972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "5783ed67079141e0b87577dd7b0e308a"
format_stamp: "Formatted at 2026-09-28 08:56:00 on dist-test-slave-pbjp"
server_key: "49136c1969d042b1b168347403f46c0c"
server_key_iv: "e7dc0b14884c10856f70576ab9e3a1a8"
server_key_version: "kuduclusterkey@0"
I20260928 08:56:00.274637 3972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "5783ed67079141e0b87577dd7b0e308a"
format_stamp: "Formatted at 2026-09-28 08:56:00 on dist-test-slave-pbjp"
server_key: "49136c1969d042b1b168347403f46c0c"
server_key_iv: "e7dc0b14884c10856f70576ab9e3a1a8"
server_key_version: "kuduclusterkey@0"
I20260928 08:56:00.276437 3972 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 08:56:00.292886 3987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:00.293174 3972 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 08:56:00.293295 3972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "5783ed67079141e0b87577dd7b0e308a"
format_stamp: "Formatted at 2026-09-28 08:56:00 on dist-test-slave-pbjp"
server_key: "49136c1969d042b1b168347403f46c0c"
server_key_iv: "e7dc0b14884c10856f70576ab9e3a1a8"
server_key_version: "kuduclusterkey@0"
I20260928 08:56:00.293401 3972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:56:00.313992 3972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:56:00.316606 3972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:56:00.316852 3972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:56:00.321861 3972 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:40389
I20260928 08:56:00.322376 3972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:56:00.322837 4039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:40389 every 8 connection(s)
I20260928 08:56:00.323261 4040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:56:00.324846 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 3972
I20260928 08:56:00.324954 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:56:00.325217 24223 external_mini_cluster.cc:1428] Setting key 6339463343fa689b9b421e5e29de4626
I20260928 08:56:00.326292 4040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a: Bootstrap starting.
I20260928 08:56:00.326960 4040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a: Neither blocks nor log segments found. Creating new log.
I20260928 08:56:00.327287 4040 log.cc:824] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a: Log is configured to *not* fsync() on all Append() calls
I20260928 08:56:00.328039 4040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a: No bootstrap required, opened a new log
I20260928 08:56:00.330127 4040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5783ed67079141e0b87577dd7b0e308a" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 40389 } }
I20260928 08:56:00.330312 4040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:56:00.330354 4040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5783ed67079141e0b87577dd7b0e308a, State: Initialized, Role: FOLLOWER
I20260928 08:56:00.330505 4040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5783ed67079141e0b87577dd7b0e308a" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 40389 } }
I20260928 08:56:00.330585 4040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:56:00.330613 4040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:56:00.330651 4040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:56:00.331327 4040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5783ed67079141e0b87577dd7b0e308a" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 40389 } }
I20260928 08:56:00.331435 4040 leader_election.cc:302] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5783ed67079141e0b87577dd7b0e308a; no voters:
I20260928 08:56:00.331633 4040 leader_election.cc:288] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:56:00.331671 4045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:56:00.331844 4045 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [term 1 LEADER]: Becoming Leader. State: Replica: 5783ed67079141e0b87577dd7b0e308a, State: Running, Role: LEADER
I20260928 08:56:00.331933 4040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:56:00.331955 4045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5783ed67079141e0b87577dd7b0e308a" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 40389 } }
I20260928 08:56:00.332273 4045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5783ed67079141e0b87577dd7b0e308a. Latest consensus state: current_term: 1 leader_uuid: "5783ed67079141e0b87577dd7b0e308a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5783ed67079141e0b87577dd7b0e308a" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 40389 } } }
I20260928 08:56:00.332346 4045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [sys.catalog]: This master's current role is: LEADER
I20260928 08:56:00.332284 4046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5783ed67079141e0b87577dd7b0e308a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5783ed67079141e0b87577dd7b0e308a" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 40389 } } }
I20260928 08:56:00.332530 4046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [sys.catalog]: This master's current role is: LEADER
I20260928 08:56:00.333652 4040 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 28 08:56:00 dist-test-slave-pbjp krb5kdc[3528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:56:00.337309 4051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:56:00.337893 4051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:56:00.341202 4051 catalog_manager.cc:1347] Generated new cluster ID: 0448230a23b44c8a8aeff70818abe177
I20260928 08:56:00.341452 4051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:56:00.345803 4043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:00.325770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53924 (local address 127.23.167.254:40389)
0928 08:56:00.326062 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:00.326074 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:00.326107 (+ 33us) server_negotiation.cc:407] Connection header received
0928 08:56:00.326662 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:00.326672 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:00.326739 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:00.326887 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:00.327866 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:00.329255 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:00.330459 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:00.331086 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:00.331172 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:00.335676 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:00.335708 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:00.335709 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:00.335764 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:00.343829 (+ 8065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:00.344245 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:00.344249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:00.344249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:00.344338 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:00.344498 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:00.344498 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:00.344499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:00.344812 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:00.344939 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:00.345454 (+ 515us) server_negotiation.cc:299] Negotiation successful
0928 08:56:00.345589 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":63,"threads_started":1}
I20260928 08:56:00.351817 4051 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:56:00.352561 4051 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:56:00.364003 4051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a: Generated new TSK 0
I20260928 08:56:00.364306 4051 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.167.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
I20260928 08:56:01.192397 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.167.212:46241/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:0
--local_ip_for_outbound_sockets=127.23.167.193
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:40389
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:39765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:56:01.312752 4085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:56:01.313145 4085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:56:01.313225 4085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:56:01.315212 4085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:56:01.315352 4085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:56:01.315455 4085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:56:01.317966 4085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:39765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.167.212:46241/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:40389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:56:01.318485 4085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:56:01.318818 4085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:56:01.327617 4093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:01.327668 4090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:01.327948 4091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:01.328270 4085 server_base.cc:1034] running on GCE node
I20260928 08:56:01.328483 4085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:56:01.328742 4085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:56:01.329905 4085 hybrid_clock.cc:648] HybridClock initialized: now 1790585761329878 us; error 42 us; skew 500 ppm
Sep 28 08:56:01 dist-test-slave-pbjp krb5kdc[3528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585761, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:01.333173 4085 init.cc:377] Logged in from keytab as kudu/127.23.167.193@KRBTEST.COM (short username kudu)
I20260928 08:56:01.333722 4085 webserver.cc:460] Webserver started at http://127.23.167.193:35515/ using document root <none> and password file <none>
I20260928 08:56:01.334028 4085 fs_manager.cc:359] Metadata directory not provided
I20260928 08:56:01.334122 4085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:56:01.334283 4085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 08:56:01 dist-test-slave-pbjp krb5kdc[3528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585761, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for HTTP/127.23.167.212@KRBTEST.COM
I20260928 08:56:01.357283 4085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-0/data/instance:
uuid: "01a6a87584fe4de9befc6e0a82d0e919"
format_stamp: "Formatted at 2026-09-28 08:56:01 on dist-test-slave-pbjp"
server_key: "9b21031a44cf016e6c969a35668d97ff"
server_key_iv: "c18bca00827cbd7c82c115473ec319b0"
server_key_version: "kuduclusterkey@0"
I20260928 08:56:01.357750 4085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance:
uuid: "01a6a87584fe4de9befc6e0a82d0e919"
format_stamp: "Formatted at 2026-09-28 08:56:01 on dist-test-slave-pbjp"
server_key: "9b21031a44cf016e6c969a35668d97ff"
server_key_iv: "c18bca00827cbd7c82c115473ec319b0"
server_key_version: "kuduclusterkey@0"
I20260928 08:56:01.359751 4085 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 08:56:01.373700 4102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:01.374009 4085 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:56:01.374190 4085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "01a6a87584fe4de9befc6e0a82d0e919"
format_stamp: "Formatted at 2026-09-28 08:56:01 on dist-test-slave-pbjp"
server_key: "9b21031a44cf016e6c969a35668d97ff"
server_key_iv: "c18bca00827cbd7c82c115473ec319b0"
server_key_version: "kuduclusterkey@0"
I20260928 08:56:01.374588 4085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:56:01.405301 4085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:56:01.407302 4085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:56:01.407452 4085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:56:01.407684 4085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:56:01.408037 4085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:56:01.408077 4085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:01.408107 4085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:56:01.408144 4085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:01.414530 4085 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:33111
I20260928 08:56:01.415066 4085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
I20260928 08:56:01.416716 4215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:33111 every 8 connection(s)
I20260928 08:56:01.423036 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 4085
I20260928 08:56:01.423149 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance
I20260928 08:56:01.423345 24223 external_mini_cluster.cc:1428] Setting key b10b29306ee52b4446bcb01f4ca7bdd5
Sep 28 08:56:01 dist-test-slave-pbjp krb5kdc[3528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585761, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:56:01.429677 4218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:01.417822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:40389 (local address 127.23.167.193:40331)
0928 08:56:01.418286 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:01.418336 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:01.418994 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:01.419525 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:01.419530 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:01.419775 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:56:01.420066 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:01.420073 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:01.421103 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:01.421104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:01.422591 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:01.422595 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:01.423287 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:01.423288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:01.423493 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:01.424340 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:56:01.424351 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:56:01.426013 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:56:01.427947 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:01.427954 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:01.427957 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:01.428250 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:01.428473 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:01.428474 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:01.428474 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:01.428606 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:01.428883 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:56:01.428889 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:56:01.429146 (+ 257us) client_negotiation.cc:769] Sending connection context
0928 08:56:01.429260 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 08:56:01.429416 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":83,"threads_started":1}
I20260928 08:56:01.430234 4219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:01.418117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:40331 (local address 127.23.167.254:40389)
0928 08:56:01.419210 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:01.419213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:01.419221 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:56:01.419247 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:01.419248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:01.419261 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:01.419318 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:01.420176 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:01.420976 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:01.422689 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:01.423017 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:01.423077 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:01.426123 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:01.426124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:01.426124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:01.426132 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:01.427854 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:01.428332 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:01.428334 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:01.428334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:01.428398 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:01.428670 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:01.428671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:01.428671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:01.428804 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:01.428859 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:01.430046 (+ 1187us) server_negotiation.cc:299] Negotiation successful
0928 08:56:01.430135 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1069,"thread_start_us":53,"threads_started":1}
I20260928 08:56:01.430964 4216 heartbeater.cc:347] Connected to a master server at 127.23.167.254:40389
I20260928 08:56:01.431061 4216 heartbeater.cc:464] Registering TS with master...
I20260928 08:56:01.431252 4216 heartbeater.cc:511] Master 127.23.167.254:40389 requested a full tablet report, sending...
I20260928 08:56:01.431905 4004 ts_manager.cc:194] Registered new tserver with Master: 01a6a87584fe4de9befc6e0a82d0e919 (127.23.167.193:33111)
I20260928 08:56:01.432757 4004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.193@KRBTEST.COM'} at 127.23.167.193:40331
WARNING: no policy specified for kudu/127.23.167.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
I20260928 08:56:01.468201 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.167.212:46241/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:0
--local_ip_for_outbound_sockets=127.23.167.194
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:40389
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:39765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:56:01 dist-test-slave-pbjp krb5kdc[3528](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790585761, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 08:56:01.604907 4224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:56:01.605298 4224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:56:01.605382 4224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:56:01.607144 4224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:56:01.607372 4224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:56:01.607477 4224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:56:01.609846 4224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:39765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.167.212:46241/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:40389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:56:01.610277 4224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:56:01.610601 4224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:56:01.618036 4232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:01.618173 4230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:01.618036 4229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:01.618352 4224 server_base.cc:1034] running on GCE node
I20260928 08:56:01.618552 4224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:56:01.618803 4224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:56:01.619956 4224 hybrid_clock.cc:648] HybridClock initialized: now 1790585761619685 us; error 283 us; skew 500 ppm
Sep 28 08:56:01 dist-test-slave-pbjp krb5kdc[3528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585761, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:01.622866 4224 init.cc:377] Logged in from keytab as kudu/127.23.167.194@KRBTEST.COM (short username kudu)
I20260928 08:56:01.623399 4224 webserver.cc:460] Webserver started at http://127.23.167.194:45683/ using document root <none> and password file <none>
I20260928 08:56:01.623642 4224 fs_manager.cc:359] Metadata directory not provided
I20260928 08:56:01.623698 4224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:56:01.623827 4224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 08:56:01 dist-test-slave-pbjp krb5kdc[3528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585761, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for HTTP/127.23.167.212@KRBTEST.COM
I20260928 08:56:01.639437 4224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-1/data/instance:
uuid: "09f470f2c15a436eab5e27185799d0b1"
format_stamp: "Formatted at 2026-09-28 08:56:01 on dist-test-slave-pbjp"
server_key: "de4e27587f26b6153bc4d56acfb6257a"
server_key_iv: "fac9e4fa6e28601cf5973f2d6ec6c890"
server_key_version: "kuduclusterkey@0"
I20260928 08:56:01.639899 4224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance:
uuid: "09f470f2c15a436eab5e27185799d0b1"
format_stamp: "Formatted at 2026-09-28 08:56:01 on dist-test-slave-pbjp"
server_key: "de4e27587f26b6153bc4d56acfb6257a"
server_key_iv: "fac9e4fa6e28601cf5973f2d6ec6c890"
server_key_version: "kuduclusterkey@0"
I20260928 08:56:01.645474 4224 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260928 08:56:01.661228 4240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:01.661485 4224 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 08:56:01.661605 4224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "09f470f2c15a436eab5e27185799d0b1"
format_stamp: "Formatted at 2026-09-28 08:56:01 on dist-test-slave-pbjp"
server_key: "de4e27587f26b6153bc4d56acfb6257a"
server_key_iv: "fac9e4fa6e28601cf5973f2d6ec6c890"
server_key_version: "kuduclusterkey@0"
I20260928 08:56:01.661715 4224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:56:01.688867 4224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:56:01.691269 4224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:56:01.691483 4224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:56:01.691784 4224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:56:01.692216 4224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:56:01.692307 4224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:01.692374 4224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:56:01.692426 4224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:01.699290 4224 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:39257
I20260928 08:56:01.699833 4224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
I20260928 08:56:01.702993 4353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:39257 every 8 connection(s)
I20260928 08:56:01.706598 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 4224
I20260928 08:56:01.706707 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance
I20260928 08:56:01.706895 24223 external_mini_cluster.cc:1428] Setting key f4640d72550c9c3f11eeff40e59c0f50
WARNING: no policy specified for kudu/127.23.167.195@KRBTEST.COM; defaulting to no policy
Sep 28 08:56:01 dist-test-slave-pbjp krb5kdc[3528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585761, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:56:01.731119 4219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:01.702974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:48509 (local address 127.23.167.254:40389)
0928 08:56:01.703037 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:01.703039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:01.708882 (+ 5843us) server_negotiation.cc:407] Connection header received
0928 08:56:01.715221 (+ 6339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:01.715224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:01.715244 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:01.715277 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:01.716191 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:01.716975 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:01.722779 (+ 5804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:01.723125 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:01.723182 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:01.727738 (+ 4556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:01.727739 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:01.727740 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:01.727748 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:01.729491 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:01.730028 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:01.730029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:01.730030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:01.730089 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:01.730360 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:01.730361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:01.730361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:01.730499 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:01.730556 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:01.730929 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 08:56:01.731019 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 08:56:01.731945 4356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:01.702683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:40389 (local address 127.23.167.194:48509)
0928 08:56:01.708699 (+ 6016us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:01.708782 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:01.715059 (+ 6277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:01.715507 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:01.715512 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:01.715784 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:56:01.716080 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:01.716087 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:01.717117 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:01.717118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:01.718434 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:01.718437 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:01.723301 (+ 4864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:01.723302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:01.723461 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:01.724142 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:56:01.724153 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:56:01.727614 (+ 3461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:56:01.729606 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:01.729613 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:01.729616 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:01.729934 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:01.730151 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:01.730151 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:01.730152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:01.730293 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:01.730591 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:56:01.730598 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:56:01.730774 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 08:56:01.731467 (+ 693us) client_negotiation.cc:240] Negotiation successful
0928 08:56:01.731658 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5951,"thread_start_us":91,"threads_started":1}
I20260928 08:56:01.732774 4354 heartbeater.cc:347] Connected to a master server at 127.23.167.254:40389
I20260928 08:56:01.732884 4354 heartbeater.cc:464] Registering TS with master...
I20260928 08:56:01.733104 4354 heartbeater.cc:511] Master 127.23.167.254:40389 requested a full tablet report, sending...
I20260928 08:56:01.733523 4004 ts_manager.cc:194] Registered new tserver with Master: 09f470f2c15a436eab5e27185799d0b1 (127.23.167.194:39257)
I20260928 08:56:01.734027 4004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.194@KRBTEST.COM'} at 127.23.167.194:48509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
I20260928 08:56:01.756659 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.167.212:46241/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:0
--local_ip_for_outbound_sockets=127.23.167.195
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:40389
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:39765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:56:01.877398 4359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:56:01.877605 4359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:56:01.877636 4359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:56:01.878787 4359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:56:01.878829 4359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:56:01.878862 4359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:56:01.880314 4359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:39765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.167.212:46241/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:40389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:56:01.880530 4359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:56:01.880748 4359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:56:01.885381 4364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:01.885767 4365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:01.885854 4359 server_base.cc:1034] running on GCE node
W20260928 08:56:01.886251 4367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:01.886453 4359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:56:01.886715 4359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:56:01.887849 4359 hybrid_clock.cc:648] HybridClock initialized: now 1790585761887838 us; error 24 us; skew 500 ppm
Sep 28 08:56:01 dist-test-slave-pbjp krb5kdc[3528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585761, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:01.890324 4359 init.cc:377] Logged in from keytab as kudu/127.23.167.195@KRBTEST.COM (short username kudu)
I20260928 08:56:01.890713 4359 webserver.cc:460] Webserver started at http://127.23.167.195:46265/ using document root <none> and password file <none>
I20260928 08:56:01.890918 4359 fs_manager.cc:359] Metadata directory not provided
I20260928 08:56:01.890965 4359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:56:01.891100 4359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 08:56:01 dist-test-slave-pbjp krb5kdc[3528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585761, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for HTTP/127.23.167.212@KRBTEST.COM
I20260928 08:56:01.912668 4359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-2/data/instance:
uuid: "92747351f28545f5b725060fbfd64898"
format_stamp: "Formatted at 2026-09-28 08:56:01 on dist-test-slave-pbjp"
server_key: "aeb9c917472ec1930ac6c308412c1453"
server_key_iv: "0a1a2a8e618388cbecd1ddcd5466e3a8"
server_key_version: "kuduclusterkey@0"
I20260928 08:56:01.913230 4359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance:
uuid: "92747351f28545f5b725060fbfd64898"
format_stamp: "Formatted at 2026-09-28 08:56:01 on dist-test-slave-pbjp"
server_key: "aeb9c917472ec1930ac6c308412c1453"
server_key_iv: "0a1a2a8e618388cbecd1ddcd5466e3a8"
server_key_version: "kuduclusterkey@0"
I20260928 08:56:01.917219 4359 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.000s
I20260928 08:56:01.937311 4374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:01.937669 4359 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260928 08:56:01.937791 4359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "92747351f28545f5b725060fbfd64898"
format_stamp: "Formatted at 2026-09-28 08:56:01 on dist-test-slave-pbjp"
server_key: "aeb9c917472ec1930ac6c308412c1453"
server_key_iv: "0a1a2a8e618388cbecd1ddcd5466e3a8"
server_key_version: "kuduclusterkey@0"
I20260928 08:56:01.937898 4359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:56:01.955291 4359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:56:01.957965 4359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:56:01.958187 4359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:56:01.958464 4359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:56:01.958868 4359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:56:01.958918 4359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:01.958954 4359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:56:01.958976 4359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:01.968102 4359 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:37689
I20260928 08:56:01.968137 4487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:37689 every 8 connection(s)
I20260928 08:56:01.968555 4359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
I20260928 08:56:01.971263 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 4359
I20260928 08:56:01.971395 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance
I20260928 08:56:01.971565 24223 external_mini_cluster.cc:1428] Setting key 8493e33d6d04ebb920ece9226b063e79
Sep 28 08:56:01 dist-test-slave-pbjp krb5kdc[3528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585761, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:56:01.981156 4219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:01.969844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:51027 (local address 127.23.167.254:40389)
0928 08:56:01.969893 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:01.969895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:01.970101 (+ 206us) server_negotiation.cc:407] Connection header received
0928 08:56:01.970831 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:01.970833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:01.970845 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:01.970875 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:01.971740 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:01.972542 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:01.974690 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:01.975014 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:01.975059 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:01.977706 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:01.977708 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:01.977708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:01.977716 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:01.979465 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:01.979973 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:01.979975 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:01.979975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:01.980038 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:01.980340 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:01.980342 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:01.980342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:01.980505 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:01.980549 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:01.980935 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 08:56:01.981025 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 08:56:01.982029 4490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:01.969572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:40389 (local address 127.23.167.195:51027)
0928 08:56:01.969970 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:01.970016 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:01.970700 (+ 684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:01.971063 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:01.971067 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:01.971313 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:56:01.971624 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:01.971631 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:01.973275 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:01.973276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:01.974589 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:01.974592 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:01.975160 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:01.975161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:01.975300 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:01.975866 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:56:01.975877 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:56:01.977581 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:56:01.979564 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:01.979572 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:01.979574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:01.979870 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:01.980110 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:01.980111 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:01.980111 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:01.980259 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:01.980597 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:56:01.980604 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:56:01.980777 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 08:56:01.981537 (+ 760us) client_negotiation.cc:240] Negotiation successful
0928 08:56:01.981717 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":76,"threads_started":1}
I20260928 08:56:01.982810 4488 heartbeater.cc:347] Connected to a master server at 127.23.167.254:40389
I20260928 08:56:01.982909 4488 heartbeater.cc:464] Registering TS with master...
I20260928 08:56:01.983108 4488 heartbeater.cc:511] Master 127.23.167.254:40389 requested a full tablet report, sending...
I20260928 08:56:01.984149 4004 ts_manager.cc:194] Registered new tserver with Master: 92747351f28545f5b725060fbfd64898 (127.23.167.195:37689)
I20260928 08:56:01.984668 4004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.195@KRBTEST.COM'} at 127.23.167.195:51027
I20260928 08:56:01.994760 24223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:56:01.995666 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 4085
I20260928 08:56:02.004216 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 4224
I20260928 08:56:02.011607 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 4359
I20260928 08:56:02.018165 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 3972
I20260928 08:56:02.023869 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:40389
--webserver_interface=127.23.167.254
--webserver_port=33125
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:39765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:40389
--ranger_config_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.167.212:46241/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.167.212:46241/kms with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:56:02.124922 4491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:56:02.125197 4491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:56:02.125236 4491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:56:02.126408 4491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:56:02.126454 4491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:56:02.126473 4491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:56:02.126489 4491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:56:02.126503 4491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:56:02.128105 4491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:39765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.167.212:46241/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:40389
--ranger_config_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:40389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=33125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:56:02.128348 4491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:56:02.128557 4491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:56:02.133474 4496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:02.133441 4499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:02.133567 4491 server_base.cc:1034] running on GCE node
W20260928 08:56:02.133416 4497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:02.133872 4491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:56:02.134176 4491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:56:02.135329 4491 hybrid_clock.cc:648] HybridClock initialized: now 1790585762135320 us; error 28 us; skew 500 ppm
Sep 28 08:56:02 dist-test-slave-pbjp krb5kdc[3528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585762, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:02.137917 4491 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:56:02.138405 4491 webserver.cc:460] Webserver started at http://127.23.167.254:33125/ using document root <none> and password file <none>
I20260928 08:56:02.138605 4491 fs_manager.cc:359] Metadata directory not provided
I20260928 08:56:02.138660 4491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 08:56:02 dist-test-slave-pbjp krb5kdc[3528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585762, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for HTTP/127.23.167.212@KRBTEST.COM
I20260928 08:56:02.187299 4491 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:56:02.188120 4507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:02.188337 4491 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:56:02.188452 4491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "5783ed67079141e0b87577dd7b0e308a"
format_stamp: "Formatted at 2026-09-28 08:56:00 on dist-test-slave-pbjp"
server_key: "49136c1969d042b1b168347403f46c0c"
server_key_iv: "e7dc0b14884c10856f70576ab9e3a1a8"
server_key_version: "kuduclusterkey@0"
I20260928 08:56:02.188925 4491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:56:02.240783 4491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:56:02.242980 4491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:56:02.243136 4491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:56:02.248121 4491 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:40389
I20260928 08:56:02.248157 4559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:40389 every 8 connection(s)
I20260928 08:56:02.248652 4491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:56:02.249969 4560 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:56:02.250656 4560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a: Bootstrap starting.
I20260928 08:56:02.250797 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 4491
I20260928 08:56:02.251091 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.167.212:46241/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:33111
--local_ip_for_outbound_sockets=127.23.167.193
--tserver_master_addrs=127.23.167.254:40389
--webserver_port=35515
--webserver_interface=127.23.167.193
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:39765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.167.212:46241/kms with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:56:02.258721 4560 log.cc:824] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a: Log is configured to *not* fsync() on all Append() calls
I20260928 08:56:02.260519 4560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:56:02.260792 4560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a: Bootstrap complete.
I20260928 08:56:02.263044 4560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5783ed67079141e0b87577dd7b0e308a" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 40389 } }
I20260928 08:56:02.263301 4560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5783ed67079141e0b87577dd7b0e308a, State: Initialized, Role: FOLLOWER
I20260928 08:56:02.263460 4560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5783ed67079141e0b87577dd7b0e308a" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 40389 } }
I20260928 08:56:02.263535 4560 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:56:02.263584 4560 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:56:02.263639 4560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [term 1 FOLLOWER]: Advancing to term 2
I20260928 08:56:02.264631 4560 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5783ed67079141e0b87577dd7b0e308a" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 40389 } }
I20260928 08:56:02.264726 4560 leader_election.cc:302] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5783ed67079141e0b87577dd7b0e308a; no voters:
I20260928 08:56:02.264899 4560 leader_election.cc:288] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 08:56:02.264948 4564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [term 2 FOLLOWER]: Leader election won for term 2
I20260928 08:56:02.265054 4564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [term 2 LEADER]: Becoming Leader. State: Replica: 5783ed67079141e0b87577dd7b0e308a, State: Running, Role: LEADER
I20260928 08:56:02.265185 4560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:56:02.265185 4564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5783ed67079141e0b87577dd7b0e308a" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 40389 } }
I20260928 08:56:02.265635 4565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5783ed67079141e0b87577dd7b0e308a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5783ed67079141e0b87577dd7b0e308a" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 40389 } } }
I20260928 08:56:02.265642 4566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5783ed67079141e0b87577dd7b0e308a. Latest consensus state: current_term: 2 leader_uuid: "5783ed67079141e0b87577dd7b0e308a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5783ed67079141e0b87577dd7b0e308a" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 40389 } } }
I20260928 08:56:02.265729 4565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [sys.catalog]: This master's current role is: LEADER
I20260928 08:56:02.265733 4566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a [sys.catalog]: This master's current role is: LEADER
I20260928 08:56:02.265913 4570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:56:02.266394 4570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:56:02.266525 4570 catalog_manager.cc:1259] Loaded cluster ID: 0448230a23b44c8a8aeff70818abe177
I20260928 08:56:02.266559 4570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:56:02.266760 4560 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 08:56:02.267737 4570 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:56:02.267917 4570 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a: Loaded TSK: 0
I20260928 08:56:02.268136 4570 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 08:56:02.367072 4562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:56:02.367368 4562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:56:02.367439 4562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:56:02.369426 4562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:56:02.369496 4562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:56:02.369540 4562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:56:02.371766 4562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:39765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.167.212:46241/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:33111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=35515
--tserver_master_addrs=127.23.167.254:40389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:56:02.372031 4562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:56:02.372324 4562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:56:02.378943 4597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:02.379046 4600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:02.378962 4598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:02.380139 4562 server_base.cc:1034] running on GCE node
I20260928 08:56:02.380420 4562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:56:02.380702 4562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:56:02.381873 4562 hybrid_clock.cc:648] HybridClock initialized: now 1790585762381837 us; error 49 us; skew 500 ppm
Sep 28 08:56:02 dist-test-slave-pbjp krb5kdc[3528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585762, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:02.385195 4562 init.cc:377] Logged in from keytab as kudu/127.23.167.193@KRBTEST.COM (short username kudu)
I20260928 08:56:02.385733 4562 webserver.cc:460] Webserver started at http://127.23.167.193:35515/ using document root <none> and password file <none>
I20260928 08:56:02.386011 4562 fs_manager.cc:359] Metadata directory not provided
I20260928 08:56:02.386092 4562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 08:56:02 dist-test-slave-pbjp krb5kdc[3528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585762, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for HTTP/127.23.167.212@KRBTEST.COM
I20260928 08:56:02.415978 4562 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:56:02.417057 4608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:02.417244 4562 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:56:02.417322 4562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "01a6a87584fe4de9befc6e0a82d0e919"
format_stamp: "Formatted at 2026-09-28 08:56:01 on dist-test-slave-pbjp"
server_key: "9b21031a44cf016e6c969a35668d97ff"
server_key_iv: "c18bca00827cbd7c82c115473ec319b0"
server_key_version: "kuduclusterkey@0"
I20260928 08:56:02.417804 4562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:56:02.440533 4562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:56:02.442870 4562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:56:02.443037 4562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:56:02.443271 4562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:56:02.443651 4562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:56:02.443712 4562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:02.443750 4562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:56:02.443771 4562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:02.450207 4562 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:33111
I20260928 08:56:02.450722 4562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
I20260928 08:56:02.453679 4721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:33111 every 8 connection(s)
I20260928 08:56:02.457204 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 4562
I20260928 08:56:02.457489 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.167.212:46241/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:39257
--local_ip_for_outbound_sockets=127.23.167.194
--tserver_master_addrs=127.23.167.254:40389
--webserver_port=45683
--webserver_interface=127.23.167.194
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:39765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.167.212:46241/kms with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:56:02 dist-test-slave-pbjp krb5kdc[3528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585762, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:56:02.464887 4724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:02.454301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:40389 (local address 127.23.167.193:57677)
0928 08:56:02.454470 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:02.454513 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:02.455058 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:02.455507 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:02.455510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:02.455696 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:56:02.455883 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:02.455888 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:02.456972 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:02.456973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:02.457999 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:02.458002 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:02.458579 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:02.458580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:02.458705 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:02.459276 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:56:02.459286 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:56:02.460779 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:56:02.462937 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:02.462944 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:02.462947 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:02.463244 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:02.463523 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:02.463524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:02.463525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:02.463652 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:02.464160 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:56:02.464166 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:56:02.464320 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 08:56:02.464427 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 08:56:02.464643 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
I20260928 08:56:02.465866 4725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:02.454567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:57677 (local address 127.23.167.254:40389)
0928 08:56:02.454771 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:02.454781 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:02.454814 (+ 33us) server_negotiation.cc:407] Connection header received
0928 08:56:02.455288 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:02.455292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:02.455337 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:02.455449 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:02.455996 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:02.456849 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:02.458081 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:02.458481 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:02.458575 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:02.460889 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:02.460894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:02.460894 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:02.460907 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:02.462845 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:02.463336 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:02.463339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:02.463340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:02.463450 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:02.463737 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:02.463737 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:02.463738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:02.464040 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:02.464129 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:02.465490 (+ 1361us) server_negotiation.cc:299] Negotiation successful
0928 08:56:02.465656 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":73,"threads_started":1}
I20260928 08:56:02.466740 4722 heartbeater.cc:347] Connected to a master server at 127.23.167.254:40389
I20260928 08:56:02.466832 4722 heartbeater.cc:464] Registering TS with master...
I20260928 08:56:02.467017 4722 heartbeater.cc:511] Master 127.23.167.254:40389 requested a full tablet report, sending...
W20260928 08:56:02.467878 4722 heartbeater.cc:644] Failed to heartbeat to 127.23.167.254:40389 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 08:56:02.467944 4722 heartbeater.cc:464] Registering TS with master...
I20260928 08:56:02.468006 4722 heartbeater.cc:511] Master 127.23.167.254:40389 requested a full tablet report, sending...
I20260928 08:56:02.468417 4722 heartbeater.cc:464] Registering TS with master...
I20260928 08:56:02.468487 4722 heartbeater.cc:511] Master 127.23.167.254:40389 requested a full tablet report, sending...
W20260928 08:56:02.468818 4722 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 08:56:02.560338 4726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:56:02.560618 4726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:56:02.560655 4726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:56:02.562368 4726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:56:02.562419 4726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:56:02.562457 4726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:56:02.564572 4726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:39765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.167.212:46241/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:39257
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=45683
--tserver_master_addrs=127.23.167.254:40389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:56:02.564877 4726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:56:02.565163 4726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:56:02.570155 4732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:02.570206 4731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:02.570315 4726 server_base.cc:1034] running on GCE node
W20260928 08:56:02.570175 4734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:02.570580 4726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:56:02.570789 4726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:56:02.571936 4726 hybrid_clock.cc:648] HybridClock initialized: now 1790585762571925 us; error 25 us; skew 500 ppm
Sep 28 08:56:02 dist-test-slave-pbjp krb5kdc[3528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585762, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:02.574395 4726 init.cc:377] Logged in from keytab as kudu/127.23.167.194@KRBTEST.COM (short username kudu)
I20260928 08:56:02.574860 4726 webserver.cc:460] Webserver started at http://127.23.167.194:45683/ using document root <none> and password file <none>
I20260928 08:56:02.575083 4726 fs_manager.cc:359] Metadata directory not provided
I20260928 08:56:02.575145 4726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 08:56:02 dist-test-slave-pbjp krb5kdc[3528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585762, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for HTTP/127.23.167.212@KRBTEST.COM
I20260928 08:56:02.591027 4726 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:56:02.592038 4742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:02.592262 4726 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:56:02.592355 4726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "09f470f2c15a436eab5e27185799d0b1"
format_stamp: "Formatted at 2026-09-28 08:56:01 on dist-test-slave-pbjp"
server_key: "de4e27587f26b6153bc4d56acfb6257a"
server_key_iv: "fac9e4fa6e28601cf5973f2d6ec6c890"
server_key_version: "kuduclusterkey@0"
I20260928 08:56:02.592810 4726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:56:02.607481 4726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:56:02.609630 4726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:56:02.609802 4726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:56:02.610059 4726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:56:02.610419 4726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:56:02.610460 4726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:02.610491 4726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:56:02.610507 4726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:02.616957 4726 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:39257
I20260928 08:56:02.617116 4855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:39257 every 8 connection(s)
I20260928 08:56:02.617470 4726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
Sep 28 08:56:02 dist-test-slave-pbjp krb5kdc[3528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585762, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:56:02.625689 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 4726
I20260928 08:56:02.625962 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.167.212:46241/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:37689
--local_ip_for_outbound_sockets=127.23.167.195
--tserver_master_addrs=127.23.167.254:40389
--webserver_port=46265
--webserver_interface=127.23.167.195
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:39765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.167.212:46241/kms with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:56:02.627969 4725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:02.618474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:35645 (local address 127.23.167.254:40389)
0928 08:56:02.618714 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:02.618717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:02.618862 (+ 145us) server_negotiation.cc:407] Connection header received
0928 08:56:02.619487 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:02.619489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:02.619505 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:02.619576 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:02.620316 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:02.620819 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:02.622060 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:02.622289 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:02.622348 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:02.624733 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:02.624735 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:02.624735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:02.624744 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:02.626318 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:02.626824 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:02.626825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:02.626826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:02.626891 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:02.627170 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:02.627171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:02.627171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:02.627317 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:02.627387 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:02.627777 (+ 390us) server_negotiation.cc:299] Negotiation successful
0928 08:56:02.627872 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208}
I20260928 08:56:02.628533 4858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:02.618478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:40389 (local address 127.23.167.194:35645)
0928 08:56:02.618668 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:02.618721 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:02.619350 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:02.619733 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:02.619738 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:02.619968 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:56:02.620204 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:02.620209 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:02.620917 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:02.620919 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:02.621965 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:02.621968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:02.622381 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:02.622383 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:02.622512 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:02.623088 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:56:02.623096 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:56:02.624609 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:56:02.626431 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:02.626438 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:02.626441 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:02.626740 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:02.626962 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:02.626962 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:02.626963 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:02.627094 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:02.627420 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:56:02.627427 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:56:02.627605 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 08:56:02.628123 (+ 518us) client_negotiation.cc:240] Negotiation successful
0928 08:56:02.628296 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":79,"threads_started":1}
I20260928 08:56:02.628968 4856 heartbeater.cc:347] Connected to a master server at 127.23.167.254:40389
I20260928 08:56:02.629115 4856 heartbeater.cc:464] Registering TS with master...
I20260928 08:56:02.629338 4856 heartbeater.cc:511] Master 127.23.167.254:40389 requested a full tablet report, sending...
W20260928 08:56:02.629978 4856 heartbeater.cc:644] Failed to heartbeat to 127.23.167.254:40389 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 08:56:02.630054 4856 heartbeater.cc:464] Registering TS with master...
I20260928 08:56:02.630128 4856 heartbeater.cc:511] Master 127.23.167.254:40389 requested a full tablet report, sending...
I20260928 08:56:02.630409 4856 heartbeater.cc:464] Registering TS with master...
I20260928 08:56:02.630482 4856 heartbeater.cc:511] Master 127.23.167.254:40389 requested a full tablet report, sending...
W20260928 08:56:02.630760 4856 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 08:56:02.722494 4859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:56:02.722704 4859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:56:02.722735 4859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:56:02.724334 4859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:56:02.724399 4859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:56:02.724438 4859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:56:02.726112 4859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:39765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.167.212:46241/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:37689
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=46265
--tserver_master_addrs=127.23.167.254:40389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:56:02.726348 4859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:56:02.726625 4859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:56:02.732892 4867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:02.732882 4865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:02.732885 4864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:02.733351 4859 server_base.cc:1034] running on GCE node
I20260928 08:56:02.733565 4859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:56:02.733868 4859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:56:02.735036 4859 hybrid_clock.cc:648] HybridClock initialized: now 1790585762735019 us; error 33 us; skew 500 ppm
Sep 28 08:56:02 dist-test-slave-pbjp krb5kdc[3528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585762, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:02.739573 4859 init.cc:377] Logged in from keytab as kudu/127.23.167.195@KRBTEST.COM (short username kudu)
I20260928 08:56:02.740209 4859 webserver.cc:460] Webserver started at http://127.23.167.195:46265/ using document root <none> and password file <none>
I20260928 08:56:02.740522 4859 fs_manager.cc:359] Metadata directory not provided
I20260928 08:56:02.740674 4859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 08:56:02 dist-test-slave-pbjp krb5kdc[3528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585762, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for HTTP/127.23.167.212@KRBTEST.COM
I20260928 08:56:02.759917 4859 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.001s
I20260928 08:56:02.761495 4876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:02.761866 4859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260928 08:56:02.762035 4859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "92747351f28545f5b725060fbfd64898"
format_stamp: "Formatted at 2026-09-28 08:56:01 on dist-test-slave-pbjp"
server_key: "aeb9c917472ec1930ac6c308412c1453"
server_key_iv: "0a1a2a8e618388cbecd1ddcd5466e3a8"
server_key_version: "kuduclusterkey@0"
I20260928 08:56:02.762694 4859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:56:02.783571 4859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:56:02.785532 4859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:56:02.785652 4859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:56:02.785903 4859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:56:02.786293 4859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:56:02.786340 4859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:02.786372 4859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:56:02.786397 4859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:02.792943 4859 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:37689
I20260928 08:56:02.793107 4989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:37689 every 8 connection(s)
I20260928 08:56:02.793402 4859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
Sep 28 08:56:02 dist-test-slave-pbjp krb5kdc[3528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585762, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:56:02.802402 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 4859
I20260928 08:56:02.808773 4725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:02.794160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:48799 (local address 127.23.167.254:40389)
0928 08:56:02.794221 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:02.794223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:02.794458 (+ 235us) server_negotiation.cc:407] Connection header received
0928 08:56:02.794963 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:02.794965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:02.794978 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:02.795025 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:02.795654 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:02.796240 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:02.797789 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:02.797984 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:02.798056 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:02.805540 (+ 7484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:02.805542 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:02.805542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:02.805561 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:02.807194 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:02.807720 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:02.807721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:02.807722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:02.807790 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:02.808051 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:02.808052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:02.808053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:02.808186 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:02.808247 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:02.808602 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 08:56:02.808682 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 08:56:02.809012 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:02.794169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:40389 (local address 127.23.167.195:48799)
0928 08:56:02.794327 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:02.794364 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:02.794840 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:02.795096 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:02.795099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:02.795323 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:56:02.795547 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:02.795553 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:02.796355 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:02.796357 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:02.797694 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:02.797697 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:02.799741 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:02.799742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:02.799850 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:02.800528 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:56:02.800537 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:56:02.805415 (+ 4878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:56:02.807293 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:02.807302 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:02.807305 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:02.807627 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:02.807855 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:02.807855 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:02.807856 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:02.807968 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:02.808273 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:56:02.808280 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:56:02.808456 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 08:56:02.808564 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 08:56:02.808719 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
I20260928 08:56:02.809515 4990 heartbeater.cc:347] Connected to a master server at 127.23.167.254:40389
I20260928 08:56:02.809615 4990 heartbeater.cc:464] Registering TS with master...
I20260928 08:56:02.809798 4990 heartbeater.cc:511] Master 127.23.167.254:40389 requested a full tablet report, sending...
W20260928 08:56:02.810519 4990 heartbeater.cc:644] Failed to heartbeat to 127.23.167.254:40389 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 08:56:02.810597 4990 heartbeater.cc:464] Registering TS with master...
I20260928 08:56:02.810665 4990 heartbeater.cc:511] Master 127.23.167.254:40389 requested a full tablet report, sending...
I20260928 08:56:02.810952 4990 heartbeater.cc:464] Registering TS with master...
I20260928 08:56:02.811028 4990 heartbeater.cc:511] Master 127.23.167.254:40389 requested a full tablet report, sending...
W20260928 08:56:02.811365 4990 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 08:56:02.813016 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:02.802863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51444 (local address 127.23.167.254:40389)
0928 08:56:02.803002 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:02.803004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:02.805294 (+ 2290us) server_negotiation.cc:407] Connection header received
0928 08:56:02.805407 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:02.805409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:02.805424 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:02.805466 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:02.806287 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:02.807072 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:02.808093 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:02.808384 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:02.808443 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:02.810509 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:02.810511 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:02.810511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:02.810517 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:02.811997 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:02.812321 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:02.812322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:02.812322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:02.812377 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:02.812540 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:02.812541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:02.812541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:02.812654 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:02.812806 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:02.812840 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 08:56:02.812915 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
Sep 28 08:56:03 dist-test-slave-pbjp krb5kdc[3528](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790585763, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:03.469007 4722 heartbeater.cc:464] Registering TS with master...
I20260928 08:56:03.469198 4722 heartbeater.cc:511] Master 127.23.167.254:40389 requested a full tablet report, sending...
I20260928 08:56:03.469626 4524 ts_manager.cc:194] Registered new tserver with Master: 01a6a87584fe4de9befc6e0a82d0e919 (127.23.167.193:33111)
I20260928 08:56:03.470681 4524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.193@KRBTEST.COM'} at 127.23.167.193:57677
I20260928 08:56:03.630937 4856 heartbeater.cc:464] Registering TS with master...
I20260928 08:56:03.631096 4856 heartbeater.cc:511] Master 127.23.167.254:40389 requested a full tablet report, sending...
I20260928 08:56:03.631404 4524 ts_manager.cc:194] Registered new tserver with Master: 09f470f2c15a436eab5e27185799d0b1 (127.23.167.194:39257)
I20260928 08:56:03.631954 4524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.194@KRBTEST.COM'} at 127.23.167.194:35645
Sep 28 08:56:03 dist-test-slave-pbjp krb5kdc[3528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585763, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for HTTP/127.23.167.212@KRBTEST.COM
I20260928 08:56:03.811537 4990 heartbeater.cc:464] Registering TS with master...
I20260928 08:56:03.811697 4990 heartbeater.cc:511] Master 127.23.167.254:40389 requested a full tablet report, sending...
I20260928 08:56:03.812044 4524 ts_manager.cc:194] Registered new tserver with Master: 92747351f28545f5b725060fbfd64898 (127.23.167.195:37689)
I20260928 08:56:03.812534 4524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.195@KRBTEST.COM'} at 127.23.167.195:48799
I20260928 08:56:03.816454 24223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:56:03.830500 5011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:03.821343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51450 (local address 127.23.167.254:40389)
0928 08:56:03.822671 (+ 1328us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:03.822673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:03.822680 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:56:03.822703 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:03.822704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:03.822716 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:03.823516 (+ 800us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:03.823554 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:03.824355 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:03.825451 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:03.825778 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:03.826117 (+ 339us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:03.827310 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:03.827311 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:03.827311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:03.827318 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:03.829016 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:03.829446 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:03.829447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:03.829448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:03.829507 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:03.829701 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:03.829701 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:03.829702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:03.829829 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:03.830274 (+ 445us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:03.830314 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 08:56:03.830394 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1309,"thread_start_us":65,"threads_started":1}
I20260928 08:56:03.832978 4524 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:56:03.835284 4524 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:56:03.845170 5020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:03.839132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:39257 (local address 127.0.0.1:45348)
0928 08:56:03.840459 (+ 1327us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:03.840467 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:03.840503 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:03.840849 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:03.840852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:03.840854 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:56:03.840922 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:03.840924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:03.842596 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:03.842597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:03.843816 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:03.843819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:03.844863 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:03.844864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:03.844944 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:03.844958 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 08:56:03.845019 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 08:56:03.845095 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1314,"thread_start_us":54,"threads_started":1}
I20260928 08:56:03.845477 5021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:03.839312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45348 (local address 127.23.167.194:39257)
0928 08:56:03.840616 (+ 1304us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:03.840621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:03.840631 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:56:03.840669 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:03.840671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:03.840689 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:03.840756 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:56:03.841009 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:03.842513 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:03.843912 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:03.844705 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:03.844774 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:03.844831 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:03.845349 (+ 518us) server_negotiation.cc:299] Negotiation successful
0928 08:56:03.845392 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1271,"thread_start_us":49,"threads_started":1}
I20260928 08:56:03.846094 4790 tablet_service.cc:1467] Processing CreateTablet for tablet c75dc7edac8641d09b2f0d9ed1f0d941 (DEFAULT_TABLE table=test-table [id=2390d6fb093a4efda93097396c0fe254]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:56:03.846465 4790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c75dc7edac8641d09b2f0d9ed1f0d941. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:56:03.848695 5018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:03.839516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:37689 (local address 127.0.0.1:47586)
0928 08:56:03.843937 (+ 4421us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:03.843946 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:03.843988 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:03.844495 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:03.844496 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:03.844499 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:56:03.844538 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:03.844540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:03.846055 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:03.846057 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:03.847521 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:03.847523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:03.848473 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:03.848474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:03.848533 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:03.848537 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:56:03.848583 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 08:56:03.848625 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4397,"thread_start_us":79,"threads_started":1}
I20260928 08:56:03.849053 5023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:03.839755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47586 (local address 127.23.167.195:37689)
0928 08:56:03.840414 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:03.840419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:03.844274 (+ 3855us) server_negotiation.cc:407] Connection header received
0928 08:56:03.844313 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:03.844315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:03.844332 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:03.844383 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:56:03.844605 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:03.845971 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:03.847611 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:03.848316 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:03.848384 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:03.848437 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:03.848927 (+ 490us) server_negotiation.cc:299] Negotiation successful
0928 08:56:03.848979 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":58,"threads_started":1}
I20260928 08:56:03.849885 4924 tablet_service.cc:1467] Processing CreateTablet for tablet c75dc7edac8641d09b2f0d9ed1f0d941 (DEFAULT_TABLE table=test-table [id=2390d6fb093a4efda93097396c0fe254]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:56:03.850399 4924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c75dc7edac8641d09b2f0d9ed1f0d941. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:56:03.853125 5022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:03.838921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:33111 (local address 127.0.0.1:49598)
0928 08:56:03.840170 (+ 1249us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:03.840215 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:03.840277 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:03.849407 (+ 9130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:03.849408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:03.849411 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:56:03.849451 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:03.849453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:03.850711 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:03.850713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:03.851823 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:03.851826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:03.852839 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:03.852840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:03.852912 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:03.852916 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:56:03.852964 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 08:56:03.853007 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1217,"thread_start_us":82,"threads_started":1}
I20260928 08:56:03.853482 5019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:03.839017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49598 (local address 127.23.167.193:33111)
0928 08:56:03.849169 (+ 10152us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:03.849174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:03.849186 (+ 12us) server_negotiation.cc:407] Connection header received
0928 08:56:03.849221 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:03.849223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:03.849239 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:03.849320 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:56:03.849536 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:03.850613 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:03.851928 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:03.852671 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:03.852745 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:03.852800 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:03.853351 (+ 551us) server_negotiation.cc:299] Negotiation successful
0928 08:56:03.853403 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10113,"thread_start_us":65,"threads_started":1}
I20260928 08:56:03.854354 5025 tablet_bootstrap.cc:492] T c75dc7edac8641d09b2f0d9ed1f0d941 P 09f470f2c15a436eab5e27185799d0b1: Bootstrap starting.
I20260928 08:56:03.854383 4656 tablet_service.cc:1467] Processing CreateTablet for tablet c75dc7edac8641d09b2f0d9ed1f0d941 (DEFAULT_TABLE table=test-table [id=2390d6fb093a4efda93097396c0fe254]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:56:03.854755 4656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c75dc7edac8641d09b2f0d9ed1f0d941. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:56:03.855248 5025 tablet_bootstrap.cc:654] T c75dc7edac8641d09b2f0d9ed1f0d941 P 09f470f2c15a436eab5e27185799d0b1: Neither blocks nor log segments found. Creating new log.
I20260928 08:56:03.855527 5025 log.cc:824] T c75dc7edac8641d09b2f0d9ed1f0d941 P 09f470f2c15a436eab5e27185799d0b1: Log is configured to *not* fsync() on all Append() calls
I20260928 08:56:03.856274 5025 tablet_bootstrap.cc:492] T c75dc7edac8641d09b2f0d9ed1f0d941 P 09f470f2c15a436eab5e27185799d0b1: No bootstrap required, opened a new log
I20260928 08:56:03.856338 5025 ts_tablet_manager.cc:1397] T c75dc7edac8641d09b2f0d9ed1f0d941 P 09f470f2c15a436eab5e27185799d0b1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:56:03.857281 5024 tablet_bootstrap.cc:492] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898: Bootstrap starting.
I20260928 08:56:03.858008 5024 tablet_bootstrap.cc:654] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898: Neither blocks nor log segments found. Creating new log.
I20260928 08:56:03.858382 5024 log.cc:824] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898: Log is configured to *not* fsync() on all Append() calls
I20260928 08:56:03.858923 5025 raft_consensus.cc:348] T c75dc7edac8641d09b2f0d9ed1f0d941 P 09f470f2c15a436eab5e27185799d0b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a6a87584fe4de9befc6e0a82d0e919" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 33111 } } peers { permanent_uuid: "92747351f28545f5b725060fbfd64898" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 37689 } } peers { permanent_uuid: "09f470f2c15a436eab5e27185799d0b1" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 39257 } }
I20260928 08:56:03.859067 5025 raft_consensus.cc:374] T c75dc7edac8641d09b2f0d9ed1f0d941 P 09f470f2c15a436eab5e27185799d0b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:56:03.859099 5025 raft_consensus.cc:729] T c75dc7edac8641d09b2f0d9ed1f0d941 P 09f470f2c15a436eab5e27185799d0b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f470f2c15a436eab5e27185799d0b1, State: Initialized, Role: FOLLOWER
I20260928 08:56:03.859232 5025 consensus_queue.cc:260] T c75dc7edac8641d09b2f0d9ed1f0d941 P 09f470f2c15a436eab5e27185799d0b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a6a87584fe4de9befc6e0a82d0e919" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 33111 } } peers { permanent_uuid: "92747351f28545f5b725060fbfd64898" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 37689 } } peers { permanent_uuid: "09f470f2c15a436eab5e27185799d0b1" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 39257 } }
I20260928 08:56:03.859493 5025 ts_tablet_manager.cc:1428] T c75dc7edac8641d09b2f0d9ed1f0d941 P 09f470f2c15a436eab5e27185799d0b1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:56:03.859624 4856 heartbeater.cc:503] Master 127.23.167.254:40389 was elected leader, sending a full tablet report...
I20260928 08:56:03.860719 5024 tablet_bootstrap.cc:492] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898: No bootstrap required, opened a new log
I20260928 08:56:03.860795 5024 ts_tablet_manager.cc:1397] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 08:56:03.861267 5027 tablet_bootstrap.cc:492] T c75dc7edac8641d09b2f0d9ed1f0d941 P 01a6a87584fe4de9befc6e0a82d0e919: Bootstrap starting.
I20260928 08:56:03.861989 5027 tablet_bootstrap.cc:654] T c75dc7edac8641d09b2f0d9ed1f0d941 P 01a6a87584fe4de9befc6e0a82d0e919: Neither blocks nor log segments found. Creating new log.
I20260928 08:56:03.862365 5027 log.cc:824] T c75dc7edac8641d09b2f0d9ed1f0d941 P 01a6a87584fe4de9befc6e0a82d0e919: Log is configured to *not* fsync() on all Append() calls
I20260928 08:56:03.863191 5024 raft_consensus.cc:348] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a6a87584fe4de9befc6e0a82d0e919" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 33111 } } peers { permanent_uuid: "92747351f28545f5b725060fbfd64898" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 37689 } } peers { permanent_uuid: "09f470f2c15a436eab5e27185799d0b1" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 39257 } }
I20260928 08:56:03.863324 5024 raft_consensus.cc:374] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:56:03.863354 5024 raft_consensus.cc:729] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92747351f28545f5b725060fbfd64898, State: Initialized, Role: FOLLOWER
I20260928 08:56:03.863478 5024 consensus_queue.cc:260] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a6a87584fe4de9befc6e0a82d0e919" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 33111 } } peers { permanent_uuid: "92747351f28545f5b725060fbfd64898" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 37689 } } peers { permanent_uuid: "09f470f2c15a436eab5e27185799d0b1" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 39257 } }
I20260928 08:56:03.863737 4990 heartbeater.cc:503] Master 127.23.167.254:40389 was elected leader, sending a full tablet report...
I20260928 08:56:03.863730 5024 ts_tablet_manager.cc:1428] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 08:56:03.869652 5027 tablet_bootstrap.cc:492] T c75dc7edac8641d09b2f0d9ed1f0d941 P 01a6a87584fe4de9befc6e0a82d0e919: No bootstrap required, opened a new log
I20260928 08:56:03.869740 5027 ts_tablet_manager.cc:1397] T c75dc7edac8641d09b2f0d9ed1f0d941 P 01a6a87584fe4de9befc6e0a82d0e919: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260928 08:56:03.871330 5027 raft_consensus.cc:348] T c75dc7edac8641d09b2f0d9ed1f0d941 P 01a6a87584fe4de9befc6e0a82d0e919 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a6a87584fe4de9befc6e0a82d0e919" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 33111 } } peers { permanent_uuid: "92747351f28545f5b725060fbfd64898" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 37689 } } peers { permanent_uuid: "09f470f2c15a436eab5e27185799d0b1" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 39257 } }
I20260928 08:56:03.871466 5027 raft_consensus.cc:374] T c75dc7edac8641d09b2f0d9ed1f0d941 P 01a6a87584fe4de9befc6e0a82d0e919 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:56:03.871497 5027 raft_consensus.cc:729] T c75dc7edac8641d09b2f0d9ed1f0d941 P 01a6a87584fe4de9befc6e0a82d0e919 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01a6a87584fe4de9befc6e0a82d0e919, State: Initialized, Role: FOLLOWER
I20260928 08:56:03.871614 5027 consensus_queue.cc:260] T c75dc7edac8641d09b2f0d9ed1f0d941 P 01a6a87584fe4de9befc6e0a82d0e919 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a6a87584fe4de9befc6e0a82d0e919" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 33111 } } peers { permanent_uuid: "92747351f28545f5b725060fbfd64898" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 37689 } } peers { permanent_uuid: "09f470f2c15a436eab5e27185799d0b1" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 39257 } }
I20260928 08:56:03.871891 5027 ts_tablet_manager.cc:1428] T c75dc7edac8641d09b2f0d9ed1f0d941 P 01a6a87584fe4de9befc6e0a82d0e919: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260928 08:56:03.872063 4857 tablet.cc:2369] T c75dc7edac8641d09b2f0d9ed1f0d941 P 09f470f2c15a436eab5e27185799d0b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:56:03.871905 4722 heartbeater.cc:503] Master 127.23.167.254:40389 was elected leader, sending a full tablet report...
Sep 28 08:56:03 dist-test-slave-pbjp krb5kdc[3528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585763, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for HTTP/127.23.167.212@KRBTEST.COM
W20260928 08:56:03.954731 4723 tablet.cc:2369] T c75dc7edac8641d09b2f0d9ed1f0d941 P 01a6a87584fe4de9befc6e0a82d0e919: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:56:04.044636 4991 tablet.cc:2369] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:56:04.171574 5031 raft_consensus.cc:482] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:56:04.171715 5031 raft_consensus.cc:504] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a6a87584fe4de9befc6e0a82d0e919" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 33111 } } peers { permanent_uuid: "92747351f28545f5b725060fbfd64898" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 37689 } } peers { permanent_uuid: "09f470f2c15a436eab5e27185799d0b1" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 39257 } }
I20260928 08:56:04.172106 5031 leader_election.cc:288] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01a6a87584fe4de9befc6e0a82d0e919 (127.23.167.193:33111), 09f470f2c15a436eab5e27185799d0b1 (127.23.167.194:39257)
I20260928 08:56:04.177734 5019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:04.172710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:43901 (local address 127.23.167.193:33111)
0928 08:56:04.172761 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:04.172763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:04.172999 (+ 236us) server_negotiation.cc:407] Connection header received
0928 08:56:04.173105 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:04.173107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:04.173118 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:04.173171 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:56:04.173378 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:04.174146 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:04.176656 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:04.177319 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:04.177371 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:04.177382 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:04.177608 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 08:56:04.177655 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 08:56:04.178220 5036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:04.172299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:33111 (local address 127.23.167.195:43901)
0928 08:56:04.172933 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:04.172944 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:04.173016 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:04.173222 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:04.173223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:04.173225 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:56:04.173306 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:04.173307 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:04.174247 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:04.174248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:04.176542 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:04.176545 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:04.177458 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:04.177459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:04.177530 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:04.177534 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:56:04.178071 (+ 537us) client_negotiation.cc:240] Negotiation successful
0928 08:56:04.178118 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":1170,"thread_start_us":103,"threads_started":1}
I20260928 08:56:04.178727 4674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75dc7edac8641d09b2f0d9ed1f0d941" candidate_uuid: "92747351f28545f5b725060fbfd64898" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01a6a87584fe4de9befc6e0a82d0e919" is_pre_election: true
I20260928 08:56:04.178920 4674 raft_consensus.cc:2454] T c75dc7edac8641d09b2f0d9ed1f0d941 P 01a6a87584fe4de9befc6e0a82d0e919 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92747351f28545f5b725060fbfd64898 in term 0.
I20260928 08:56:04.179173 4877 leader_election.cc:302] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01a6a87584fe4de9befc6e0a82d0e919, 92747351f28545f5b725060fbfd64898; no voters:
I20260928 08:56:04.179297 5031 raft_consensus.cc:2781] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:56:04.179339 5031 raft_consensus.cc:482] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:56:04.179358 5031 raft_consensus.cc:3037] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:56:04.179838 5021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:04.172765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:47083 (local address 127.23.167.194:39257)
0928 08:56:04.172804 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:04.172805 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:04.173199 (+ 394us) server_negotiation.cc:407] Connection header received
0928 08:56:04.173285 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:04.173286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:04.173294 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:04.173320 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:56:04.173489 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:04.174169 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:04.178636 (+ 4467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:04.179218 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:04.179273 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:04.179281 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:04.179726 (+ 445us) server_negotiation.cc:299] Negotiation successful
0928 08:56:04.179771 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 08:56:04.180298 5031 raft_consensus.cc:504] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a6a87584fe4de9befc6e0a82d0e919" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 33111 } } peers { permanent_uuid: "92747351f28545f5b725060fbfd64898" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 37689 } } peers { permanent_uuid: "09f470f2c15a436eab5e27185799d0b1" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 39257 } }
I20260928 08:56:04.180454 5031 leader_election.cc:288] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898 [CANDIDATE]: Term 1 election: Requested vote from peers 01a6a87584fe4de9befc6e0a82d0e919 (127.23.167.193:33111), 09f470f2c15a436eab5e27185799d0b1 (127.23.167.194:39257)
I20260928 08:56:04.180471 5035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:04.172499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:39257 (local address 127.23.167.195:47083)
0928 08:56:04.173151 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:04.173160 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:04.173221 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:04.173376 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:04.173377 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:04.173378 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:56:04.173424 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:04.173425 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:04.174247 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:04.174248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:04.175519 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:04.175524 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:04.179563 (+ 4039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:04.179564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:04.179639 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:04.179643 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:56:04.180101 (+ 458us) client_negotiation.cc:240] Negotiation successful
0928 08:56:04.180146 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":77,"thread_start_us":56,"threads_started":1}
I20260928 08:56:04.180651 4674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75dc7edac8641d09b2f0d9ed1f0d941" candidate_uuid: "92747351f28545f5b725060fbfd64898" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01a6a87584fe4de9befc6e0a82d0e919"
I20260928 08:56:04.180725 4674 raft_consensus.cc:3037] T c75dc7edac8641d09b2f0d9ed1f0d941 P 01a6a87584fe4de9befc6e0a82d0e919 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:56:04.180938 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75dc7edac8641d09b2f0d9ed1f0d941" candidate_uuid: "92747351f28545f5b725060fbfd64898" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f470f2c15a436eab5e27185799d0b1" is_pre_election: true
I20260928 08:56:04.181137 4810 raft_consensus.cc:2454] T c75dc7edac8641d09b2f0d9ed1f0d941 P 09f470f2c15a436eab5e27185799d0b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92747351f28545f5b725060fbfd64898 in term 0.
I20260928 08:56:04.181282 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75dc7edac8641d09b2f0d9ed1f0d941" candidate_uuid: "92747351f28545f5b725060fbfd64898" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f470f2c15a436eab5e27185799d0b1"
I20260928 08:56:04.181344 4809 raft_consensus.cc:3037] T c75dc7edac8641d09b2f0d9ed1f0d941 P 09f470f2c15a436eab5e27185799d0b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:56:04.181602 4674 raft_consensus.cc:2454] T c75dc7edac8641d09b2f0d9ed1f0d941 P 01a6a87584fe4de9befc6e0a82d0e919 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92747351f28545f5b725060fbfd64898 in term 1.
I20260928 08:56:04.181778 4877 leader_election.cc:302] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01a6a87584fe4de9befc6e0a82d0e919, 92747351f28545f5b725060fbfd64898; no voters:
I20260928 08:56:04.181865 5031 raft_consensus.cc:2781] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:56:04.182009 5031 raft_consensus.cc:686] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898 [term 1 LEADER]: Becoming Leader. State: Replica: 92747351f28545f5b725060fbfd64898, State: Running, Role: LEADER
I20260928 08:56:04.182114 4809 raft_consensus.cc:2454] T c75dc7edac8641d09b2f0d9ed1f0d941 P 09f470f2c15a436eab5e27185799d0b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92747351f28545f5b725060fbfd64898 in term 1.
I20260928 08:56:04.182102 5031 consensus_queue.cc:237] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a6a87584fe4de9befc6e0a82d0e919" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 33111 } } peers { permanent_uuid: "92747351f28545f5b725060fbfd64898" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 37689 } } peers { permanent_uuid: "09f470f2c15a436eab5e27185799d0b1" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 39257 } }
I20260928 08:56:04.182909 4524 catalog_manager.cc:5495] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92747351f28545f5b725060fbfd64898 (127.23.167.195). New cstate: current_term: 1 leader_uuid: "92747351f28545f5b725060fbfd64898" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a6a87584fe4de9befc6e0a82d0e919" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 33111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92747351f28545f5b725060fbfd64898" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 37689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09f470f2c15a436eab5e27185799d0b1" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 39257 } health_report { overall_health: UNKNOWN } } }
I20260928 08:56:04.207978 5023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:04.204280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47590 (local address 127.23.167.195:37689)
0928 08:56:04.204326 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:04.204328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:04.204336 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:56:04.204381 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:04.204382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:04.204405 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:04.204476 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:56:04.204708 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:04.205516 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:04.206791 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:04.207077 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:04.207146 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:04.207367 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:56:04.207530 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:56:04.207625 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:04.207871 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 08:56:04.207895 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 08:56:04.217769 4809 raft_consensus.cc:1261] T c75dc7edac8641d09b2f0d9ed1f0d941 P 09f470f2c15a436eab5e27185799d0b1 [term 1 FOLLOWER]: Refusing update from remote peer 92747351f28545f5b725060fbfd64898: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:56:04.217769 4674 raft_consensus.cc:1261] T c75dc7edac8641d09b2f0d9ed1f0d941 P 01a6a87584fe4de9befc6e0a82d0e919 [term 1 FOLLOWER]: Refusing update from remote peer 92747351f28545f5b725060fbfd64898: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:56:04.218164 5037 consensus_queue.cc:1035] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09f470f2c15a436eab5e27185799d0b1" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 39257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:56:04.218464 5031 consensus_queue.cc:1035] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01a6a87584fe4de9befc6e0a82d0e919" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 33111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:56:04.222664 5046 mvcc.cc:205] Tried to move back new op lower bound from 7334239290231291904 to 7334239290090373120. Current Snapshot: MvccSnapshot[applied={T|T < 7334239290231291904}]
I20260928 08:56:04.222790 5042 mvcc.cc:205] Tried to move back new op lower bound from 7334239290231291904 to 7334239290090373120. Current Snapshot: MvccSnapshot[applied={T|T < 7334239290231291904}]
I20260928 08:56:04.228999 4523 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:56:04.229141 4523 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:56:04.230090 4523 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5783ed67079141e0b87577dd7b0e308a: Sending DeleteTablet for 3 replicas of tablet c75dc7edac8641d09b2f0d9ed1f0d941
I20260928 08:56:04.230501 4656 tablet_service.cc:1514] Processing DeleteTablet for tablet c75dc7edac8641d09b2f0d9ed1f0d941 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:56:04 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:49598
I20260928 08:56:04.230870 4924 tablet_service.cc:1514] Processing DeleteTablet for tablet c75dc7edac8641d09b2f0d9ed1f0d941 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:56:04 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:47586
I20260928 08:56:04.231405 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 4562
I20260928 08:56:04.235940 4790 tablet_service.cc:1514] Processing DeleteTablet for tablet c75dc7edac8641d09b2f0d9ed1f0d941 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:56:04 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:45348
I20260928 08:56:04.237179 5057 tablet_replica.cc:330] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898: stopping tablet replica
I20260928 08:56:04.237303 5057 raft_consensus.cc:2229] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:56:04.237411 5057 raft_consensus.cc:2258] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:56:04.237881 5057 ts_tablet_manager.cc:1905] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 08:56:04.240109 4508 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.167.193:33111 (error 108)
I20260928 08:56:04.240644 5020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:04.240505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:33111 (local address 127.0.0.1:49614)
0928 08:56:04.240570 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:04.240605 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.193:33111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 08:56:04.240765 4508 catalog_manager.cc:4584] TS 01a6a87584fe4de9befc6e0a82d0e919 (127.23.167.193:33111): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c75dc7edac8641d09b2f0d9ed1f0d941: Network error: Client connection negotiation failed: client connection to 127.23.167.193:33111: connect: Connection refused (error 111)
I20260928 08:56:04.241129 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 4726
I20260928 08:56:04.241708 5057 ts_tablet_manager.cc:1918] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:56:04.241789 5057 log.cc:1196] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585589999956-24223-0/minicluster-data/ts-2/wal/wals/c75dc7edac8641d09b2f0d9ed1f0d941
I20260928 08:56:04.242065 5057 ts_tablet_manager.cc:1939] T c75dc7edac8641d09b2f0d9ed1f0d941 P 92747351f28545f5b725060fbfd64898: Deleting consensus metadata
I20260928 08:56:04.244371 4510 catalog_manager.cc:4841] TS 92747351f28545f5b725060fbfd64898 (127.23.167.195:37689): tablet c75dc7edac8641d09b2f0d9ed1f0d941 (table test-table [id=2390d6fb093a4efda93097396c0fe254]) successfully deleted
I20260928 08:56:04.248788 5020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:04.248678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:39257 (local address 127.0.0.1:45356)
0928 08:56:04.248728 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:04.248756 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.194:39257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 08:56:04.248847 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 4859
I20260928 08:56:04.254071 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 4491
I20260928 08:56:04.259305 24223 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 08:56:04.603948 24223 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 08:56:04.604063 24223 mini_ranger.cc:66] Stopping Ranger...
I20260928 08:56:04.938675 24223 mini_ranger.cc:68] Stopped Ranger
2026-09-28 08:56:04.938 UTC [3558] LOG: received smart shutdown request
2026-09-28 08:56:04.942 UTC [3558] LOG: background worker "logical replication launcher" (PID 3567) exited with exit code 1
2026-09-28 08:56:04.943 UTC [3562] LOG: shutting down
2026-09-28 08:56:05.108 UTC [3558] LOG: database system is shut down
2026-09-28T08:56:05Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47100 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:56:05 dist-test-slave-pbjp krb5kdc[5068](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:56:05 dist-test-slave-pbjp krb5kdc[5068](info): set up 2 sockets
Sep 28 08:56:05 dist-test-slave-pbjp krb5kdc[5068](info): commencing operation
krb5kdc: starting...
W20260928 08:56:06.148717 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:56:06 dist-test-slave-pbjp krb5kdc[5068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:56:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:56:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.167.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
I20260928 08:56:06.259829 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:42659
--webserver_interface=127.23.167.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:42659,127.23.167.253:42265,127.23.167.252:46393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:56:06.341671 5081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:56:06.341889 5081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:56:06.341919 5081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:56:06.343070 5081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:56:06.343111 5081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:56:06.343129 5081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:56:06.343146 5081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:56:06.343160 5081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:56:06.344669 5081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:42659,127.23.167.253:42265,127.23.167.252:46393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:42659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:56:06.344913 5081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:56:06.345148 5081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:56:06.349799 5089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:06.349799 5087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:06.349915 5081 server_base.cc:1034] running on GCE node
W20260928 08:56:06.349820 5086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:06.350271 5081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:56:06.350545 5081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:56:06.351688 5081 hybrid_clock.cc:648] HybridClock initialized: now 1790585766351674 us; error 25 us; skew 500 ppm
Sep 28 08:56:06 dist-test-slave-pbjp krb5kdc[5068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585766, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:06.354100 5081 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:56:06.354516 5081 webserver.cc:460] Webserver started at http://127.23.167.254:42345/ using document root <none> and password file <none>
I20260928 08:56:06.354784 5081 fs_manager.cc:359] Metadata directory not provided
I20260928 08:56:06.354838 5081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:56:06.354962 5081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:56:06.355993 5081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "6b73255752074a9b83d0725fab799605"
format_stamp: "Formatted at 2026-09-28 08:56:06 on dist-test-slave-pbjp"
server_key: "ed0f487c06a9f8367d0fd862b0d2f7f1"
server_key_iv: "10a359cc8218ba1b259d4861cbef9543"
server_key_version: "encryptionkey@0"
I20260928 08:56:06.356325 5081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "6b73255752074a9b83d0725fab799605"
format_stamp: "Formatted at 2026-09-28 08:56:06 on dist-test-slave-pbjp"
server_key: "ed0f487c06a9f8367d0fd862b0d2f7f1"
server_key_iv: "10a359cc8218ba1b259d4861cbef9543"
server_key_version: "encryptionkey@0"
I20260928 08:56:06.357676 5081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 08:56:06.358364 5096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:06.358544 5081 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:56:06.358649 5081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "6b73255752074a9b83d0725fab799605"
format_stamp: "Formatted at 2026-09-28 08:56:06 on dist-test-slave-pbjp"
server_key: "ed0f487c06a9f8367d0fd862b0d2f7f1"
server_key_iv: "10a359cc8218ba1b259d4861cbef9543"
server_key_version: "encryptionkey@0"
I20260928 08:56:06.358709 5081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:56:06.371407 5081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:56:06.373477 5081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:56:06.373622 5081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:56:06.377431 5081 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:42659
I20260928 08:56:06.377538 5148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:42659 every 8 connection(s)
I20260928 08:56:06.377800 5081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:56:06.378679 5149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:56:06.380232 5149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42659 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:56:06.384536 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 5081
I20260928 08:56:06.384636 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:56:06.384817 24223 external_mini_cluster.cc:1428] Setting key c72562562c83d21c5725f2489af8dddb
Sep 28 08:56:06 dist-test-slave-pbjp krb5kdc[5068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585766, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:56:06.389521 5151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.380887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42659 (local address 127.0.0.1:59216)
0928 08:56:06.381051 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:06.381124 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:06.381562 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:06.381834 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:06.381836 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:06.382074 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:56:06.382247 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:06.382251 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.383260 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.383261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:06.383939 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:06.383941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.384485 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.384486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:06.384548 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:06.385116 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:56:06.385126 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:56:06.386544 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:56:06.388144 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:06.388150 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:06.388152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:06.388323 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:06.388567 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:06.388567 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:06.388568 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:06.388642 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:06.388965 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:56:06.388969 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:56:06.389031 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 08:56:06.389137 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 08:56:06.389346 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":76,"thread_start_us":68,"threads_started":1}
I20260928 08:56:06.389525 5152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.381015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59216 (local address 127.23.167.254:42659)
0928 08:56:06.381151 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:06.381156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:06.381188 (+ 32us) server_negotiation.cc:407] Connection header received
0928 08:56:06.381701 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:06.381703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:06.381720 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:06.381814 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:06.382332 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.383186 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.384011 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.384358 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.384523 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:06.386623 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:06.386630 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:06.386630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:06.386637 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:06.388064 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:06.388389 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:06.388392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:06.388392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:06.388505 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:06.388695 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:06.388695 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:06.388696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:06.388903 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:06.388973 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:06.389160 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 08:56:06.389274 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260928 08:56:06.390412 5149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.167.253" port: 42265 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:56:06.390733 5151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.390623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:42265 (local address 127.0.0.1:59850)
0928 08:56:06.390667 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:06.390710 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.253:42265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 08:56:06.390852 5098 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.167.253:42265: connect: Connection refused (error 111)
I20260928 08:56:06.391230 5151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.391147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:42265 (local address 127.0.0.1:59852)
0928 08:56:06.391186 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:06.391210 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.253:42265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
WARNING: no policy specified for kudu/127.23.167.253@KRBTEST.COM; defaulting to no policy
W20260928 08:56:06.391322 5149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.167.253:42265: Network error: Client connection negotiation failed: client connection to 127.23.167.253:42265: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.253.keytab.
Entry for principal kudu/127.23.167.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.253.keytab.
I20260928 08:56:06.405959 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.253:42265
--webserver_interface=127.23.167.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.253.keytab
--principal=kudu/127.23.167.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:42659,127.23.167.253:42265,127.23.167.252:46393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:56:06.440547 5149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.167.253" port: 42265 } attempt: 1
I20260928 08:56:06.441030 5151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.440902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:42265 (local address 127.0.0.1:59854)
0928 08:56:06.440958 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:06.440997 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.253:42265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 08:56:06.441445 5151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.441338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:42265 (local address 127.0.0.1:59864)
0928 08:56:06.441390 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:06.441416 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.253:42265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 08:56:06.441557 5149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.167.253:42265: Network error: Client connection negotiation failed: client connection to 127.23.167.253:42265: connect: Connection refused (error 111)
W20260928 08:56:06.485190 5156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:56:06.485395 5156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:56:06.485424 5156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:56:06.486575 5156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:56:06.486614 5156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:56:06.486632 5156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:56:06.486649 5156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:56:06.486663 5156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:56:06.488101 5156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:42659,127.23.167.253:42265,127.23.167.252:46393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.253.keytab
--principal=kudu/127.23.167.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.253:42265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.167.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:56:06.488334 5156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:56:06.488540 5156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:56:06.492973 5161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:06.493119 5156 server_base.cc:1034] running on GCE node
W20260928 08:56:06.492986 5162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:06.492986 5164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:06.493463 5156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:56:06.493768 5156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:56:06.494904 5156 hybrid_clock.cc:648] HybridClock initialized: now 1790585766494880 us; error 44 us; skew 500 ppm
Sep 28 08:56:06 dist-test-slave-pbjp krb5kdc[5068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585766, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:06.497242 5156 init.cc:377] Logged in from keytab as kudu/127.23.167.253@KRBTEST.COM (short username kudu)
I20260928 08:56:06.497681 5156 webserver.cc:460] Webserver started at http://127.23.167.253:40699/ using document root <none> and password file <none>
I20260928 08:56:06.497892 5156 fs_manager.cc:359] Metadata directory not provided
I20260928 08:56:06.497943 5156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:56:06.498065 5156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:56:06.499050 5156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-1/data/instance:
uuid: "8f247207238b4fe993368eb9df4aec9d"
format_stamp: "Formatted at 2026-09-28 08:56:06 on dist-test-slave-pbjp"
server_key: "7931cd7823deb838dd3c99335482e35e"
server_key_iv: "2038dd2c70952798cc089a13eb58e380"
server_key_version: "encryptionkey@0"
I20260928 08:56:06.499368 5156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-1/wal/instance:
uuid: "8f247207238b4fe993368eb9df4aec9d"
format_stamp: "Formatted at 2026-09-28 08:56:06 on dist-test-slave-pbjp"
server_key: "7931cd7823deb838dd3c99335482e35e"
server_key_iv: "2038dd2c70952798cc089a13eb58e380"
server_key_version: "encryptionkey@0"
I20260928 08:56:06.500633 5156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 08:56:06.501305 5171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:06.501438 5156 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:06.501516 5156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-1/wal
uuid: "8f247207238b4fe993368eb9df4aec9d"
format_stamp: "Formatted at 2026-09-28 08:56:06 on dist-test-slave-pbjp"
server_key: "7931cd7823deb838dd3c99335482e35e"
server_key_iv: "2038dd2c70952798cc089a13eb58e380"
server_key_version: "encryptionkey@0"
I20260928 08:56:06.501582 5156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:56:06.509688 5149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.167.253" port: 42265 } attempt: 2
I20260928 08:56:06.510087 5151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.509969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:42265 (local address 127.0.0.1:59866)
0928 08:56:06.510021 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:06.510053 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.253:42265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 08:56:06.510401 5151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.510319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:42265 (local address 127.0.0.1:59880)
0928 08:56:06.510360 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:06.510383 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.253:42265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 08:56:06.510485 5149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.167.253:42265: Network error: Client connection negotiation failed: client connection to 127.23.167.253:42265: connect: Connection refused (error 111)
I20260928 08:56:06.520991 5156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:56:06.523206 5156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:56:06.523326 5156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:56:06.527259 5156 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.253:42265
I20260928 08:56:06.527297 5223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.253:42265 every 8 connection(s)
I20260928 08:56:06.527629 5156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-1/data/info.pb
I20260928 08:56:06.528456 5224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:56:06.530069 5224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42659 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:56:06.530378 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 5156
I20260928 08:56:06.530468 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-1/wal/instance
I20260928 08:56:06.530650 24223 external_mini_cluster.cc:1428] Setting key 531be75209f49212f716b3197ea8c974
Sep 28 08:56:06 dist-test-slave-pbjp krb5kdc[5068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585766, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.253@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.167.252@KRBTEST.COM; defaulting to no policy
I20260928 08:56:06.538866 5152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.530811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59218 (local address 127.23.167.254:42659)
0928 08:56:06.530855 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:06.530857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:06.531135 (+ 278us) server_negotiation.cc:407] Connection header received
0928 08:56:06.531591 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:06.531593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:06.531609 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:06.531679 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:06.532283 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.532767 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.533772 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.534063 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.534123 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:06.536410 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:06.536412 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:06.536412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:06.536419 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:06.537737 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:06.538105 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:06.538106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:06.538106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:06.538158 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:06.538323 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:06.538323 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:06.538323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:06.538413 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:06.538456 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:06.538730 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 08:56:06.538800 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 08:56:06.538995 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.530829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42659 (local address 127.0.0.1:59218)
0928 08:56:06.531004 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:06.531044 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:06.531487 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:06.531739 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:06.531742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:06.531985 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:56:06.532186 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:06.532190 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.532849 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.532851 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:06.533690 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:06.533693 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.534139 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.534140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:06.534260 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:06.534761 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:56:06.534770 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:56:06.536321 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:56:06.537808 (+ 1487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:06.537816 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:06.537818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:06.538033 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:06.538203 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:06.538203 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:06.538204 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:06.538288 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:06.538452 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:56:06.538455 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:56:06.538616 (+ 161us) client_negotiation.cc:769] Sending connection context
0928 08:56:06.538700 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 08:56:06.538817 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260928 08:56:06.539538 5224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.167.253" port: 42265 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.252@KRBTEST.COM" created.
Sep 28 08:56:06 dist-test-slave-pbjp krb5kdc[5068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585766, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.253@KRBTEST.COM for kudu/127.23.167.253@KRBTEST.COM
I20260928 08:56:06.546258 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.539787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:42265 (local address 127.0.0.1:59884)
0928 08:56:06.539829 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:06.539840 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:06.539880 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:06.540142 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:06.540142 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:06.540260 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:56:06.540302 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:06.540303 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.541121 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.541122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:06.541621 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:06.541623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.542048 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.542049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:06.542108 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:06.542464 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:56:06.542468 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:56:06.543726 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:56:06.545263 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:06.545264 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:06.545265 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:06.545462 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:06.545637 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:06.545638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:06.545638 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:06.545696 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:06.545920 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:56:06.545921 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:56:06.545966 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 08:56:06.546039 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 08:56:06.546132 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 08:56:06.546288 5228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.539855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59884 (local address 127.23.167.253:42265)
0928 08:56:06.539995 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:06.539999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:06.540012 (+ 13us) server_negotiation.cc:407] Connection header received
0928 08:56:06.540028 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:06.540030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:06.540043 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:06.540103 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:06.540362 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.541024 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.541686 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.541982 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.542037 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:06.543803 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:06.543809 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:06.543810 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:06.543816 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:06.545190 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:06.545538 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:06.545540 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:06.545540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:06.545585 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:06.545747 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:06.545747 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:06.545747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:06.545864 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:06.545919 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:06.546063 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 08:56:06.546175 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20260928 08:56:06.547082 5224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46393 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:56:06.547379 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.547287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:46393 (local address 127.0.0.1:46482)
0928 08:56:06.547319 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:06.547360 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.252:46393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260928 08:56:06.547513 5174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.167.252:46393: connect: Connection refused (error 111)
I20260928 08:56:06.547925 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.547848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:46393 (local address 127.0.0.1:46492)
0928 08:56:06.547879 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:06.547904 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.252:46393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260928 08:56:06.548012 5224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.167.252:46393: Network error: Client connection negotiation failed: client connection to 127.23.167.252:46393: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.252.keytab.
Entry for principal kudu/127.23.167.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.252.keytab.
I20260928 08:56:06.555480 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.252:46393
--webserver_interface=127.23.167.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.252.keytab
--principal=kudu/127.23.167.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:42659,127.23.167.253:42265,127.23.167.252:46393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:56:06.597261 5224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46393 } attempt: 1
I20260928 08:56:06.597779 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.597639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:46393 (local address 127.0.0.1:46508)
0928 08:56:06.597701 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:06.597747 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.252:46393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 08:56:06.598137 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.598040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:46393 (local address 127.0.0.1:46520)
0928 08:56:06.598081 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:06.598119 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.252:46393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 08:56:06.598241 5224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.167.252:46393: Network error: Client connection negotiation failed: client connection to 127.23.167.252:46393: connect: Connection refused (error 111)
I20260928 08:56:06.601627 5149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.167.253" port: 42265 } attempt: 3
Sep 28 08:56:06 dist-test-slave-pbjp krb5kdc[5068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585766, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for kudu/127.23.167.253@KRBTEST.COM
I20260928 08:56:06.608456 5228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.601990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59900 (local address 127.23.167.253:42265)
0928 08:56:06.602033 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:06.602035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:06.602049 (+ 14us) server_negotiation.cc:407] Connection header received
0928 08:56:06.602125 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:06.602125 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:06.602136 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:06.602196 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:06.602566 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.603089 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.604060 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.604282 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.604343 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:06.606273 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:06.606275 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:06.606275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:06.606281 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:06.607500 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:06.607851 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:06.607852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:06.607852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:06.607897 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:06.608058 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:06.608058 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:06.608059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:06.608157 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:06.608200 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:06.608331 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 08:56:06.608389 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 08:56:06.608512 5151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.601951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:42265 (local address 127.0.0.1:59900)
0928 08:56:06.602007 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:06.602018 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:06.602061 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:06.602214 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:06.602215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:06.602398 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:56:06.602488 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:06.602490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.603166 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.603167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:06.603986 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:06.603989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.604355 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.604357 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:06.604432 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:06.604877 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:56:06.604881 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:56:06.606191 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:56:06.607576 (+ 1385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:06.607577 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:06.607578 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:06.607789 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:06.607946 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:06.607947 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:06.607947 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:06.608008 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:06.608202 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:56:06.608203 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:56:06.608250 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 08:56:06.608315 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 08:56:06.608405 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 08:56:06.608881 5149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46393 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:56:06.609212 5151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.609130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:46393 (local address 127.0.0.1:46534)
0928 08:56:06.609163 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:06.609191 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.252:46393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 08:56:06.609494 5151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.609434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:46393 (local address 127.0.0.1:46546)
0928 08:56:06.609459 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:06.609480 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.252:46393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260928 08:56:06.609608 5149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.167.252:46393: Network error: Client connection negotiation failed: client connection to 127.23.167.252:46393: connect: Connection refused (error 111)
W20260928 08:56:06.636510 5231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:56:06.636749 5231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:56:06.636780 5231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:56:06.638038 5231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:56:06.638090 5231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:56:06.638111 5231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:56:06.638127 5231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:56:06.638144 5231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:56:06.639699 5231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:42659,127.23.167.253:42265,127.23.167.252:46393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.252.keytab
--principal=kudu/127.23.167.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.252:46393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.167.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:56:06.639938 5231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:56:06.640151 5231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 08:56:06.640743 5149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46393 } attempt: 1
I20260928 08:56:06.641199 5151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.641035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:46393 (local address 127.0.0.1:46552)
0928 08:56:06.641123 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:06.641164 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.252:46393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 08:56:06.641520 5151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.641437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:46393 (local address 127.0.0.1:46560)
0928 08:56:06.641468 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:06.641495 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.252:46393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260928 08:56:06.641613 5149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.167.252:46393: Network error: Client connection negotiation failed: client connection to 127.23.167.252:46393: connect: Connection refused (error 111)
W20260928 08:56:06.644908 5239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:06.644945 5231 server_base.cc:1034] running on GCE node
W20260928 08:56:06.645025 5237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:06.644917 5236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:06.645298 5231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:56:06.645548 5231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:56:06.646677 5231 hybrid_clock.cc:648] HybridClock initialized: now 1790585766646667 us; error 26 us; skew 500 ppm
Sep 28 08:56:06 dist-test-slave-pbjp krb5kdc[5068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585766, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:06.649011 5231 init.cc:377] Logged in from keytab as kudu/127.23.167.252@KRBTEST.COM (short username kudu)
I20260928 08:56:06.649451 5231 webserver.cc:460] Webserver started at http://127.23.167.252:41791/ using document root <none> and password file <none>
I20260928 08:56:06.649676 5231 fs_manager.cc:359] Metadata directory not provided
I20260928 08:56:06.649725 5231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:56:06.649849 5231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:56:06.650867 5231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-2/data/instance:
uuid: "094711e1959a45aaa8ac083362cb3974"
format_stamp: "Formatted at 2026-09-28 08:56:06 on dist-test-slave-pbjp"
server_key: "a30d520f752be9ef6dba8881f0229154"
server_key_iv: "f6575c1afe131bcb3487c9b9396c2f22"
server_key_version: "encryptionkey@0"
I20260928 08:56:06.651187 5231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-2/wal/instance:
uuid: "094711e1959a45aaa8ac083362cb3974"
format_stamp: "Formatted at 2026-09-28 08:56:06 on dist-test-slave-pbjp"
server_key: "a30d520f752be9ef6dba8881f0229154"
server_key_iv: "f6575c1afe131bcb3487c9b9396c2f22"
server_key_version: "encryptionkey@0"
I20260928 08:56:06.652417 5231 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:56:06.653175 5246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:06.653321 5231 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:56:06.653400 5231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-2/wal
uuid: "094711e1959a45aaa8ac083362cb3974"
format_stamp: "Formatted at 2026-09-28 08:56:06 on dist-test-slave-pbjp"
server_key: "a30d520f752be9ef6dba8881f0229154"
server_key_iv: "f6575c1afe131bcb3487c9b9396c2f22"
server_key_version: "encryptionkey@0"
I20260928 08:56:06.653466 5231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:56:06.666431 5224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46393 } attempt: 2
I20260928 08:56:06.666929 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.666772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:46393 (local address 127.0.0.1:46574)
0928 08:56:06.666833 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:06.666887 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.252:46393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 08:56:06.667259 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.667182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:46393 (local address 127.0.0.1:46580)
0928 08:56:06.667219 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:06.667243 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.252:46393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 08:56:06.667344 5224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.167.252:46393: Network error: Client connection negotiation failed: client connection to 127.23.167.252:46393: connect: Connection refused (error 111)
I20260928 08:56:06.680284 5231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:56:06.682456 5231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:56:06.682588 5231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:56:06.686306 5231 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.252:46393
I20260928 08:56:06.686353 5298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.252:46393 every 8 connection(s)
I20260928 08:56:06.686685 5231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-2/data/info.pb
I20260928 08:56:06.687547 5299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:56:06.689008 5299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42659 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:56:06.689982 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 5231
I20260928 08:56:06.690078 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585589999956-24223-0/minicluster-data/master-2/wal/instance
I20260928 08:56:06.690254 24223 external_mini_cluster.cc:1428] Setting key 892778255f01c3c54790a2abda08bb7e
Sep 28 08:56:06 dist-test-slave-pbjp krb5kdc[5068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585766, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.252@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
Sep 28 08:56:06 dist-test-slave-pbjp krb5kdc[5068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.252@KRBTEST.COM
I20260928 08:56:06.698491 5152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.689710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59228 (local address 127.23.167.254:42659)
0928 08:56:06.689753 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:06.689755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:06.689998 (+ 243us) server_negotiation.cc:407] Connection header received
0928 08:56:06.690618 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:06.690619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:06.690632 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:06.690688 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:06.691353 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.691866 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.692916 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.693259 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.693317 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:06.695659 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:06.695660 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:06.695660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:06.695667 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:06.697118 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:06.697491 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:06.697492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:06.697492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:06.697564 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:06.697842 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:06.697843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:06.697843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:06.698005 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:06.698060 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:06.698310 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 08:56:06.698397 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 08:56:06.698670 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.690608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46584 (local address 127.23.167.252:46393)
0928 08:56:06.690757 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:06.690761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:06.690779 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:56:06.690820 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:06.690822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:06.690840 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:06.690925 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:06.691451 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.692450 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.693310 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.693715 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.693777 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:06.696064 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:06.696072 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:06.696072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:06.696080 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:06.697523 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:06.697848 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:06.697851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:06.697851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:06.697899 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:06.698076 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:06.698076 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:06.698077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:06.698215 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:06.698350 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:06.698423 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 08:56:06.698490 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260928 08:56:06.698936 5301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.689697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42659 (local address 127.0.0.1:59228)
0928 08:56:06.689865 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:06.689913 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:06.690495 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:06.690822 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:06.690826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:06.691120 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:56:06.691264 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:06.691269 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.691934 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.691935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:06.692839 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:06.692841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.693327 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.693328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:06.693465 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:06.693952 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:56:06.693960 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:56:06.695571 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:56:06.697199 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:06.697206 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:06.697208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:06.697428 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:06.697646 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:06.697647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:06.697648 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:06.697772 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:06.698053 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:56:06.698059 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:56:06.698191 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 08:56:06.698274 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 08:56:06.698422 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":24,"thread_start_us":77,"threads_started":1}
I20260928 08:56:06.699425 5299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.167.253" port: 42265 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:56:06 dist-test-slave-pbjp krb5kdc[5068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585766, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.252@KRBTEST.COM for kudu/127.23.167.253@KRBTEST.COM
Sep 28 08:56:06 dist-test-slave-pbjp krb5kdc[5068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:56:06.706094 5301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.699638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:42265 (local address 127.0.0.1:59914)
0928 08:56:06.699684 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:06.699694 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:06.699729 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:06.699830 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:06.699831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:06.699956 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:56:06.700008 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:06.700009 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.700873 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.700874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:06.701663 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:06.701665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.702091 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.702092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:06.702160 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:06.702524 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:56:06.702528 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:56:06.703757 (+ 1229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:56:06.705257 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:06.705258 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:06.705259 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:06.705408 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:06.705572 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:06.705573 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:06.705573 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:06.705626 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:06.705851 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:56:06.705853 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:56:06.705896 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 08:56:06.705946 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 08:56:06.706010 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 08:56:06.706147 5228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.699702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59914 (local address 127.23.167.253:42265)
0928 08:56:06.699742 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:06.699744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:06.699751 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:56:06.699776 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:06.699777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:06.699786 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:06.699826 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:06.700066 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.700799 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.701734 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.702025 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.702085 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:06.703847 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:06.703848 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:06.703848 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:06.703856 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:06.705191 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:06.705467 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:06.705468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:06.705469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:06.705522 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:06.705674 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:06.705675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:06.705676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:06.705782 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:06.705839 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:06.705985 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 08:56:06.706058 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 08:56:06.706440 5299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46393 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:56:06.707154 5152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.699418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59230 (local address 127.23.167.254:42659)
0928 08:56:06.699448 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:06.699450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:06.699457 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:56:06.699513 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:06.699514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:06.699523 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:06.699565 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:06.700097 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.700802 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.701734 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.702021 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.702084 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:06.704449 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:06.704451 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:06.704451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:06.704459 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:06.705760 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:06.706072 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:06.706073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:06.706074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:06.706127 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:06.706630 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:06.706631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:06.706632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:06.706756 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:06.706950 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:06.706986 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 08:56:06.707062 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 28 08:56:06 dist-test-slave-pbjp krb5kdc[5068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585766, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.252@KRBTEST.COM for kudu/127.23.167.252@KRBTEST.COM
Sep 28 08:56:06 dist-test-slave-pbjp krb5kdc[5068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.253@KRBTEST.COM
I20260928 08:56:06.712503 5301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.706670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:46393 (local address 127.0.0.1:46600)
0928 08:56:06.706702 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:06.706711 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:06.706741 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:06.706863 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:06.706864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:06.706991 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:56:06.707020 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:06.707021 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.707667 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.707668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:06.708363 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:06.708366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.708744 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.708745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:06.708789 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:06.709169 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:56:06.709174 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:56:06.710255 (+ 1081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:56:06.711606 (+ 1351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:06.711607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:06.711608 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:06.711800 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:06.711976 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:06.711976 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:06.711977 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:06.712025 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:06.712247 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:56:06.712248 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:56:06.712294 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 08:56:06.712361 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 08:56:06.712437 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 08:56:06.712621 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.706721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46600 (local address 127.23.167.252:46393)
0928 08:56:06.706755 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:06.706757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:06.706763 (+ 6us) server_negotiation.cc:407] Connection header received
0928 08:56:06.706780 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:06.706781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:06.706790 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:06.706833 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:06.707082 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.707601 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.708438 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.708658 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.708722 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:06.710342 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:06.710343 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:06.710344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:06.710350 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:06.711508 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:06.711863 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:06.711864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:06.711864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:06.711920 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:06.712074 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:06.712075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:06.712075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:06.712182 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:06.712238 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:06.712393 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 08:56:06.712489 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":18}
I20260928 08:56:06.714320 5228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.707941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59922 (local address 127.23.167.253:42265)
0928 08:56:06.707978 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:06.707980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:06.707986 (+ 6us) server_negotiation.cc:407] Connection header received
0928 08:56:06.708047 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:06.708047 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:06.708054 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:06.708091 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:06.708582 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.709114 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.709709 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.709901 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.709951 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:06.711909 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:06.711910 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:06.711910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:06.711917 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:06.713348 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:06.713631 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:06.713631 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:06.713632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:06.713685 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:06.713839 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:06.713839 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:06.713840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:06.713956 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:06.714121 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:06.714160 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 08:56:06.714231 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 08:56:06.714587 5299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974: Bootstrap starting.
I20260928 08:56:06.715488 5299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974: Neither blocks nor log segments found. Creating new log.
I20260928 08:56:06.715785 5299 log.cc:824] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974: Log is configured to *not* fsync() on all Append() calls
I20260928 08:56:06.715931 24223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:56:06.716513 5299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974: No bootstrap required, opened a new log
I20260928 08:56:06.716747 5149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46393 } attempt: 2
I20260928 08:56:06.718564 5299 raft_consensus.cc:348] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b73255752074a9b83d0725fab799605" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42659 } } peers { permanent_uuid: "8f247207238b4fe993368eb9df4aec9d" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 42265 } } peers { permanent_uuid: "094711e1959a45aaa8ac083362cb3974" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46393 } }
I20260928 08:56:06.718717 5299 raft_consensus.cc:374] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:56:06.718746 5299 raft_consensus.cc:729] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 094711e1959a45aaa8ac083362cb3974, State: Initialized, Role: FOLLOWER
I20260928 08:56:06.718876 5299 consensus_queue.cc:260] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b73255752074a9b83d0725fab799605" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42659 } } peers { permanent_uuid: "8f247207238b4fe993368eb9df4aec9d" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 42265 } } peers { permanent_uuid: "094711e1959a45aaa8ac083362cb3974" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46393 } }
I20260928 08:56:06.719168 5299 sys_catalog.cc:564] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 08:56:06.721131 5322 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:56:06.721191 5322 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 28 08:56:06 dist-test-slave-pbjp krb5kdc[5068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585766, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for kudu/127.23.167.252@KRBTEST.COM
I20260928 08:56:06.721356 5311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b73255752074a9b83d0725fab799605" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42659 } } peers { permanent_uuid: "8f247207238b4fe993368eb9df4aec9d" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 42265 } } peers { permanent_uuid: "094711e1959a45aaa8ac083362cb3974" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46393 } } }
I20260928 08:56:06.721434 5311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:56:06.724207 5151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.717001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:46393 (local address 127.0.0.1:46606)
0928 08:56:06.717048 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:06.717058 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:06.717128 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:06.717345 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:06.717347 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:06.717509 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:56:06.717559 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:06.717561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.718425 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.718425 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:06.719155 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:06.719157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.719613 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.719615 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:06.719673 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:06.720097 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:56:06.720108 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:56:06.721526 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:56:06.723142 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:06.723144 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:06.723144 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:06.723356 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:06.723559 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:06.723560 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:06.723560 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:06.723617 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:06.723915 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:56:06.723916 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:56:06.723960 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 08:56:06.724003 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 08:56:06.724097 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 08:56:06.724582 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.717196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46606 (local address 127.23.167.252:46393)
0928 08:56:06.717234 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:06.717236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:06.717242 (+ 6us) server_negotiation.cc:407] Connection header received
0928 08:56:06.717261 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:06.717261 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:06.717270 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:06.717622 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:06.717650 (+ 28us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.718347 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.719233 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.719532 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.719837 (+ 305us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:06.721603 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:06.721604 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:06.721605 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:06.721610 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:06.723037 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:06.723438 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:06.723439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:06.723440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:06.723496 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:06.723686 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:06.723687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:06.723687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:06.723842 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:06.724386 (+ 544us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:06.724425 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 08:56:06.724498 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 08:56:06.724848 5228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.717209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59930 (local address 127.23.167.253:42265)
0928 08:56:06.717241 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:06.717243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:06.717474 (+ 231us) server_negotiation.cc:407] Connection header received
0928 08:56:06.717563 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:06.717564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:06.717573 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:06.717599 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:06.718314 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.718960 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.720261 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.720534 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.720579 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:06.722256 (+ 1677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:06.722257 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:06.722257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:06.722265 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:06.723631 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:06.724013 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:06.724014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:06.724015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:06.724074 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:06.724266 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:06.724267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:06.724267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:06.724396 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:06.724463 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:06.724685 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 08:56:06.724762 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 08:56:06.721550 5231 master_runner.cc:186] Error getting master registration for 127.23.167.254:42659: OK, instance_id { permanent_uuid: "6b73255752074a9b83d0725fab799605" instance_seqno: 1790585766376333 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 08:56:06.725749 5231 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 08:56:06.726183 5149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b73255752074a9b83d0725fab799605: Bootstrap starting.
I20260928 08:56:06.726903 5149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b73255752074a9b83d0725fab799605: Neither blocks nor log segments found. Creating new log.
I20260928 08:56:06.727227 5149 log.cc:824] T 00000000000000000000000000000000 P 6b73255752074a9b83d0725fab799605: Log is configured to *not* fsync() on all Append() calls
I20260928 08:56:06.728011 5149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b73255752074a9b83d0725fab799605: No bootstrap required, opened a new log
I20260928 08:56:06.729954 5149 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b73255752074a9b83d0725fab799605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b73255752074a9b83d0725fab799605" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42659 } } peers { permanent_uuid: "8f247207238b4fe993368eb9df4aec9d" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 42265 } } peers { permanent_uuid: "094711e1959a45aaa8ac083362cb3974" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46393 } }
I20260928 08:56:06.730091 5149 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b73255752074a9b83d0725fab799605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:56:06.730224 5152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.717207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59244 (local address 127.23.167.254:42659)
0928 08:56:06.717248 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:06.717249 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:06.717478 (+ 229us) server_negotiation.cc:407] Connection header received
0928 08:56:06.717559 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:06.717560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:06.717568 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:06.717595 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:06.718301 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.719002 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.720225 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.720487 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.720525 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:06.727213 (+ 6688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:06.727215 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:06.727215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:06.727223 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:06.728886 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:06.729298 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:06.729299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:06.729300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:06.729354 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:06.729563 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:06.729564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:06.729565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:06.729689 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:06.729773 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:06.729975 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 08:56:06.730106 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 08:56:06.730131 5149 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b73255752074a9b83d0725fab799605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b73255752074a9b83d0725fab799605, State: Initialized, Role: FOLLOWER
I20260928 08:56:06.730434 5149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b73255752074a9b83d0725fab799605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b73255752074a9b83d0725fab799605" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42659 } } peers { permanent_uuid: "8f247207238b4fe993368eb9df4aec9d" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 42265 } } peers { permanent_uuid: "094711e1959a45aaa8ac083362cb3974" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46393 } }
I20260928 08:56:06.730768 5149 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b73255752074a9b83d0725fab799605 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:56:06.733182 5327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b73255752074a9b83d0725fab799605 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b73255752074a9b83d0725fab799605" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42659 } } peers { permanent_uuid: "8f247207238b4fe993368eb9df4aec9d" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 42265 } } peers { permanent_uuid: "094711e1959a45aaa8ac083362cb3974" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46393 } } }
I20260928 08:56:06.734242 5327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b73255752074a9b83d0725fab799605 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 08:56:06.733565 5338 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b73255752074a9b83d0725fab799605: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:56:06.737192 5338 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:56:06.738016 5324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.721668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46614 (local address 127.23.167.252:46393)
0928 08:56:06.721924 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:06.721926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:06.721981 (+ 55us) server_negotiation.cc:407] Connection header received
0928 08:56:06.722371 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:06.722372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:06.722402 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:06.727975 (+ 5573us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:06.728008 (+ 33us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.728776 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.729768 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.730064 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.730138 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:06.731386 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:06.731388 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:06.731388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:06.731395 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:06.733520 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:06.733941 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:06.733942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:06.733942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:06.733997 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:06.737155 (+ 3158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:06.737156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:06.737156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:06.737305 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:06.737409 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:06.737817 (+ 408us) server_negotiation.cc:299] Negotiation successful
0928 08:56:06.737900 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":68,"threads_started":1}
I20260928 08:56:06.735152 5081 master_runner.cc:186] Error getting master registration for 127.23.167.253:42265: OK, instance_id { permanent_uuid: "8f247207238b4fe993368eb9df4aec9d" instance_seqno: 1790585766526082 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 08:56:06.738701 5081 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 08:56:06.758545 5224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46393 } attempt: 3
Sep 28 08:56:06 dist-test-slave-pbjp krb5kdc[5068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585766, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.253@KRBTEST.COM for kudu/127.23.167.252@KRBTEST.COM
I20260928 08:56:06.767131 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.758864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:46393 (local address 127.0.0.1:46618)
0928 08:56:06.758925 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:06.758936 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:06.758985 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:06.759138 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:06.759139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:06.759359 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:56:06.759429 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:06.759431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.760675 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.760676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:06.761551 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:06.761554 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.762091 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.762092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:06.762162 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:06.762625 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:56:06.762630 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:56:06.764175 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:56:06.766038 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:06.766039 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:06.766040 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:06.766264 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:06.766459 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:06.766460 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:06.766460 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:06.766527 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:06.766782 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:56:06.766783 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:56:06.766834 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 08:56:06.766904 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 08:56:06.767019 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 08:56:06.767150 5324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:06.758900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46618 (local address 127.23.167.252:46393)
0928 08:56:06.758942 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:06.758943 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:06.758972 (+ 29us) server_negotiation.cc:407] Connection header received
0928 08:56:06.759041 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:06.759042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:06.759063 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:06.759131 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:06.759501 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.760309 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.761631 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:06.762010 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:06.762094 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:06.764285 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:06.764287 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:06.764288 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:06.764295 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:06.765952 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:06.766341 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:06.766342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:06.766342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:06.766401 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:06.766584 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:06.766585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:06.766585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:06.766722 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:06.766786 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:06.766944 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 08:56:06.767042 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 08:56:06.769351 5224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f247207238b4fe993368eb9df4aec9d: Bootstrap starting.
I20260928 08:56:06.770198 5224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f247207238b4fe993368eb9df4aec9d: Neither blocks nor log segments found. Creating new log.
I20260928 08:56:06.770530 5224 log.cc:824] T 00000000000000000000000000000000 P 8f247207238b4fe993368eb9df4aec9d: Log is configured to *not* fsync() on all Append() calls
I20260928 08:56:06.771651 5224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f247207238b4fe993368eb9df4aec9d: No bootstrap required, opened a new log
I20260928 08:56:06.773662 5224 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f247207238b4fe993368eb9df4aec9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b73255752074a9b83d0725fab799605" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42659 } } peers { permanent_uuid: "8f247207238b4fe993368eb9df4aec9d" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 42265 } } peers { permanent_uuid: "094711e1959a45aaa8ac083362cb3974" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46393 } }
I20260928 08:56:06.773818 5224 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f247207238b4fe993368eb9df4aec9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:56:06.773852 5224 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f247207238b4fe993368eb9df4aec9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f247207238b4fe993368eb9df4aec9d, State: Initialized, Role: FOLLOWER
I20260928 08:56:06.773988 5224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f247207238b4fe993368eb9df4aec9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b73255752074a9b83d0725fab799605" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42659 } } peers { permanent_uuid: "8f247207238b4fe993368eb9df4aec9d" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 42265 } } peers { permanent_uuid: "094711e1959a45aaa8ac083362cb3974" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46393 } }
I20260928 08:56:06.774335 5224 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f247207238b4fe993368eb9df4aec9d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:56:06.774298 5341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f247207238b4fe993368eb9df4aec9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b73255752074a9b83d0725fab799605" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42659 } } peers { permanent_uuid: "8f247207238b4fe993368eb9df4aec9d" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 42265 } } peers { permanent_uuid: "094711e1959a45aaa8ac083362cb3974" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46393 } } }
I20260928 08:56:06.774487 5341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f247207238b4fe993368eb9df4aec9d [sys.catalog]: This master's current role is: FOLLOWER
W20260928 08:56:06.776098 5352 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8f247207238b4fe993368eb9df4aec9d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:56:06.776209 5352 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:56:06.778221 5156 master_runner.cc:263] No leader master found from master 8f247207238b4fe993368eb9df4aec9d
I20260928 08:56:06.778275 5156 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 08:56:06.874473 5102 tablet.cc:2369] T 00000000000000000000000000000000 P 6b73255752074a9b83d0725fab799605: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:56:06.931233 5311 raft_consensus.cc:482] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:56:06.931370 5311 raft_consensus.cc:504] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b73255752074a9b83d0725fab799605" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42659 } } peers { permanent_uuid: "8f247207238b4fe993368eb9df4aec9d" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 42265 } } peers { permanent_uuid: "094711e1959a45aaa8ac083362cb3974" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46393 } }
I20260928 08:56:06.931618 5311 leader_election.cc:288] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b73255752074a9b83d0725fab799605 (127.23.167.254:42659), 8f247207238b4fe993368eb9df4aec9d (127.23.167.253:42265)
I20260928 08:56:06.931885 5123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "094711e1959a45aaa8ac083362cb3974" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b73255752074a9b83d0725fab799605" is_pre_election: true
I20260928 08:56:06.932029 5123 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6b73255752074a9b83d0725fab799605 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 094711e1959a45aaa8ac083362cb3974 in term 0.
I20260928 08:56:06.932114 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "094711e1959a45aaa8ac083362cb3974" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f247207238b4fe993368eb9df4aec9d" is_pre_election: true
I20260928 08:56:06.932304 5198 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8f247207238b4fe993368eb9df4aec9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 094711e1959a45aaa8ac083362cb3974 in term 0.
I20260928 08:56:06.932252 5250 leader_election.cc:302] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 094711e1959a45aaa8ac083362cb3974, 6b73255752074a9b83d0725fab799605; no voters:
I20260928 08:56:06.932421 5311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:56:06.932477 5311 raft_consensus.cc:482] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:56:06.932499 5311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974 [term 0 FOLLOWER]: Advancing to term 1
W20260928 08:56:06.933190 5252 tablet.cc:2369] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:56:06.933290 5311 raft_consensus.cc:504] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b73255752074a9b83d0725fab799605" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42659 } } peers { permanent_uuid: "8f247207238b4fe993368eb9df4aec9d" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 42265 } } peers { permanent_uuid: "094711e1959a45aaa8ac083362cb3974" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46393 } }
I20260928 08:56:06.933427 5311 leader_election.cc:288] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974 [CANDIDATE]: Term 1 election: Requested vote from peers 6b73255752074a9b83d0725fab799605 (127.23.167.254:42659), 8f247207238b4fe993368eb9df4aec9d (127.23.167.253:42265)
I20260928 08:56:06.933600 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "094711e1959a45aaa8ac083362cb3974" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f247207238b4fe993368eb9df4aec9d"
I20260928 08:56:06.933600 5123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "094711e1959a45aaa8ac083362cb3974" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b73255752074a9b83d0725fab799605"
I20260928 08:56:06.933686 5198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f247207238b4fe993368eb9df4aec9d [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:56:06.933692 5123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b73255752074a9b83d0725fab799605 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:56:06.934571 5123 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6b73255752074a9b83d0725fab799605 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 094711e1959a45aaa8ac083362cb3974 in term 1.
I20260928 08:56:06.934573 5198 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8f247207238b4fe993368eb9df4aec9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 094711e1959a45aaa8ac083362cb3974 in term 1.
I20260928 08:56:06.934741 5250 leader_election.cc:302] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 094711e1959a45aaa8ac083362cb3974, 6b73255752074a9b83d0725fab799605; no voters:
I20260928 08:56:06.934861 5311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:56:06.935024 5311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974 [term 1 LEADER]: Becoming Leader. State: Replica: 094711e1959a45aaa8ac083362cb3974, State: Running, Role: LEADER
I20260928 08:56:06.935117 5311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b73255752074a9b83d0725fab799605" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42659 } } peers { permanent_uuid: "8f247207238b4fe993368eb9df4aec9d" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 42265 } } peers { permanent_uuid: "094711e1959a45aaa8ac083362cb3974" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46393 } }
I20260928 08:56:06.935520 5354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 094711e1959a45aaa8ac083362cb3974. Latest consensus state: current_term: 1 leader_uuid: "094711e1959a45aaa8ac083362cb3974" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b73255752074a9b83d0725fab799605" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42659 } } peers { permanent_uuid: "8f247207238b4fe993368eb9df4aec9d" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 42265 } } peers { permanent_uuid: "094711e1959a45aaa8ac083362cb3974" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46393 } } }
I20260928 08:56:06.935619 5354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974 [sys.catalog]: This master's current role is: LEADER
I20260928 08:56:06.935812 5356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:56:06.935997 5356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:56:06.937294 5198 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8f247207238b4fe993368eb9df4aec9d [term 1 FOLLOWER]: Refusing update from remote peer 094711e1959a45aaa8ac083362cb3974: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:56:06.937395 5123 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6b73255752074a9b83d0725fab799605 [term 1 FOLLOWER]: Refusing update from remote peer 094711e1959a45aaa8ac083362cb3974: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:56:06.937594 5354 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f247207238b4fe993368eb9df4aec9d" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 42265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:56:06.937685 5311 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b73255752074a9b83d0725fab799605" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:56:06.938831 5341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f247207238b4fe993368eb9df4aec9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 094711e1959a45aaa8ac083362cb3974. Latest consensus state: current_term: 1 leader_uuid: "094711e1959a45aaa8ac083362cb3974" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b73255752074a9b83d0725fab799605" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42659 } } peers { permanent_uuid: "8f247207238b4fe993368eb9df4aec9d" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 42265 } } peers { permanent_uuid: "094711e1959a45aaa8ac083362cb3974" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46393 } } }
I20260928 08:56:06.938920 5341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f247207238b4fe993368eb9df4aec9d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:56:06.939114 5327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b73255752074a9b83d0725fab799605 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 094711e1959a45aaa8ac083362cb3974. Latest consensus state: current_term: 1 leader_uuid: "094711e1959a45aaa8ac083362cb3974" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b73255752074a9b83d0725fab799605" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42659 } } peers { permanent_uuid: "8f247207238b4fe993368eb9df4aec9d" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 42265 } } peers { permanent_uuid: "094711e1959a45aaa8ac083362cb3974" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46393 } } }
I20260928 08:56:06.939209 5327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b73255752074a9b83d0725fab799605 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:56:06.939814 5311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "094711e1959a45aaa8ac083362cb3974" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b73255752074a9b83d0725fab799605" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42659 } } peers { permanent_uuid: "8f247207238b4fe993368eb9df4aec9d" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 42265 } } peers { permanent_uuid: "094711e1959a45aaa8ac083362cb3974" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46393 } } }
I20260928 08:56:06.939888 5311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974 [sys.catalog]: This master's current role is: LEADER
I20260928 08:56:06.939918 5311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "094711e1959a45aaa8ac083362cb3974" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b73255752074a9b83d0725fab799605" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42659 } } peers { permanent_uuid: "8f247207238b4fe993368eb9df4aec9d" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 42265 } } peers { permanent_uuid: "094711e1959a45aaa8ac083362cb3974" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46393 } } }
I20260928 08:56:06.939950 5311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974 [sys.catalog]: This master's current role is: LEADER
I20260928 08:56:06.939989 5357 mvcc.cc:205] Tried to move back new op lower bound from 7334239301373046784 to 7334239301366771712. Current Snapshot: MvccSnapshot[applied={T|T < 7334239301373046784}]
I20260928 08:56:06.940126 5327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b73255752074a9b83d0725fab799605 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "094711e1959a45aaa8ac083362cb3974" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b73255752074a9b83d0725fab799605" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42659 } } peers { permanent_uuid: "8f247207238b4fe993368eb9df4aec9d" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 42265 } } peers { permanent_uuid: "094711e1959a45aaa8ac083362cb3974" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46393 } } }
I20260928 08:56:06.940199 5327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b73255752074a9b83d0725fab799605 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:56:06.940419 5341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f247207238b4fe993368eb9df4aec9d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "094711e1959a45aaa8ac083362cb3974" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b73255752074a9b83d0725fab799605" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42659 } } peers { permanent_uuid: "8f247207238b4fe993368eb9df4aec9d" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 42265 } } peers { permanent_uuid: "094711e1959a45aaa8ac083362cb3974" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46393 } } }
I20260928 08:56:06.940464 5358 mvcc.cc:205] Tried to move back new op lower bound from 7334239301373046784 to 7334239301366771712. Current Snapshot: MvccSnapshot[applied={T|T < 7334239301373046784}]
I20260928 08:56:06.940493 5341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f247207238b4fe993368eb9df4aec9d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:56:06.940923 5360 mvcc.cc:205] Tried to move back new op lower bound from 7334239301373046784 to 7334239301366771712. Current Snapshot: MvccSnapshot[applied={T|T < 7334239301373046784 or (T in {7334239301373046784})}]
I20260928 08:56:06.940930 5356 catalog_manager.cc:1347] Generated new cluster ID: 005928bc0b1d4660925ebe9666ebf16e
I20260928 08:56:06.940980 5356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:56:06.990556 5356 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:56:06.991160 5356 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:56:06.995134 5356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 094711e1959a45aaa8ac083362cb3974: Generated new TSK 0
I20260928 08:56:06.995288 5356 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:56:07.737739 5338 catalog_manager.cc:1259] Loaded cluster ID: 005928bc0b1d4660925ebe9666ebf16e
I20260928 08:56:07.737826 5338 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6b73255752074a9b83d0725fab799605: loading cluster ID for follower catalog manager: success
I20260928 08:56:07.738658 5338 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6b73255752074a9b83d0725fab799605: acquiring CA information for follower catalog manager: success
I20260928 08:56:07.738960 5338 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6b73255752074a9b83d0725fab799605: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 08:56:07.776661 5352 catalog_manager.cc:1259] Loaded cluster ID: 005928bc0b1d4660925ebe9666ebf16e
I20260928 08:56:07.776737 5352 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8f247207238b4fe993368eb9df4aec9d: loading cluster ID for follower catalog manager: success
I20260928 08:56:07.777613 5352 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8f247207238b4fe993368eb9df4aec9d: acquiring CA information for follower catalog manager: success
I20260928 08:56:07.777915 5352 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8f247207238b4fe993368eb9df4aec9d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 08:56:08.060114 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 5081
I20260928 08:56:08.064988 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 5156
I20260928 08:56:08.069368 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 5231
2026-09-28T08:56:08Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2877 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:56:08 dist-test-slave-pbjp krb5kdc[5371](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:56:08 dist-test-slave-pbjp krb5kdc[5371](info): set up 2 sockets
Sep 28 08:56:08 dist-test-slave-pbjp krb5kdc[5371](info): commencing operation
krb5kdc: starting...
W20260928 08:56:09.017145 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:56:09 dist-test-slave-pbjp krb5kdc[5371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:56:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:56:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.167.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
I20260928 08:56:09.127683 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:40863
--webserver_interface=127.23.167.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:42311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:40863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:56:09.207449 5384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:56:09.207657 5384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:56:09.207688 5384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:56:09.208849 5384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:56:09.208892 5384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:56:09.208911 5384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:56:09.208926 5384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:56:09.208942 5384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:56:09.210469 5384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:42311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:40863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:40863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:56:09.210716 5384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:56:09.210922 5384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 08:56:09.215458 5384 server_base.cc:1034] running on GCE node
W20260928 08:56:09.215412 5390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:09.215436 5389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:09.215425 5392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:09.215871 5384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:56:09.216190 5384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:56:09.217327 5384 hybrid_clock.cc:648] HybridClock initialized: now 1790585769217309 us; error 30 us; skew 500 ppm
Sep 28 08:56:09 dist-test-slave-pbjp krb5kdc[5371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585769, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:09.219684 5384 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:56:09.220124 5384 webserver.cc:460] Webserver started at http://127.23.167.254:41005/ using document root <none> and password file <none>
I20260928 08:56:09.220315 5384 fs_manager.cc:359] Metadata directory not provided
I20260928 08:56:09.220362 5384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:56:09.220494 5384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:56:09.221547 5384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "688fafde05fa4ce0969f3d930d7b7b29"
format_stamp: "Formatted at 2026-09-28 08:56:09 on dist-test-slave-pbjp"
server_key: "3906091519a5b20fa5618df21fa9c0c1"
server_key_iv: "47dfc971197d38a10ba9960cdba92fae"
server_key_version: "encryptionkey@0"
I20260928 08:56:09.221861 5384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "688fafde05fa4ce0969f3d930d7b7b29"
format_stamp: "Formatted at 2026-09-28 08:56:09 on dist-test-slave-pbjp"
server_key: "3906091519a5b20fa5618df21fa9c0c1"
server_key_iv: "47dfc971197d38a10ba9960cdba92fae"
server_key_version: "encryptionkey@0"
I20260928 08:56:09.223048 5384 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:56:09.223665 5399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:09.223810 5384 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:56:09.223889 5384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "688fafde05fa4ce0969f3d930d7b7b29"
format_stamp: "Formatted at 2026-09-28 08:56:09 on dist-test-slave-pbjp"
server_key: "3906091519a5b20fa5618df21fa9c0c1"
server_key_iv: "47dfc971197d38a10ba9960cdba92fae"
server_key_version: "encryptionkey@0"
I20260928 08:56:09.223979 5384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:56:09.239919 5384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:56:09.241845 5384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:56:09.241989 5384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:56:09.245676 5384 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:40863
I20260928 08:56:09.245710 5451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:40863 every 8 connection(s)
I20260928 08:56:09.246032 5384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:56:09.246829 5452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:56:09.248981 5452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 688fafde05fa4ce0969f3d930d7b7b29: Bootstrap starting.
I20260928 08:56:09.249518 5452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 688fafde05fa4ce0969f3d930d7b7b29: Neither blocks nor log segments found. Creating new log.
I20260928 08:56:09.249744 5452 log.cc:824] T 00000000000000000000000000000000 P 688fafde05fa4ce0969f3d930d7b7b29: Log is configured to *not* fsync() on all Append() calls
I20260928 08:56:09.250376 5452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 688fafde05fa4ce0969f3d930d7b7b29: No bootstrap required, opened a new log
I20260928 08:56:09.251928 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 5384
I20260928 08:56:09.252019 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:56:09.251993 5452 raft_consensus.cc:348] T 00000000000000000000000000000000 P 688fafde05fa4ce0969f3d930d7b7b29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688fafde05fa4ce0969f3d930d7b7b29" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 40863 } }
I20260928 08:56:09.252166 5452 raft_consensus.cc:374] T 00000000000000000000000000000000 P 688fafde05fa4ce0969f3d930d7b7b29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:56:09.252208 5452 raft_consensus.cc:729] T 00000000000000000000000000000000 P 688fafde05fa4ce0969f3d930d7b7b29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 688fafde05fa4ce0969f3d930d7b7b29, State: Initialized, Role: FOLLOWER
I20260928 08:56:09.252242 24223 external_mini_cluster.cc:1428] Setting key 132c233f338f98258f4ba7d83583eaeb
I20260928 08:56:09.252352 5452 consensus_queue.cc:260] T 00000000000000000000000000000000 P 688fafde05fa4ce0969f3d930d7b7b29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688fafde05fa4ce0969f3d930d7b7b29" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 40863 } }
I20260928 08:56:09.252430 5452 raft_consensus.cc:388] T 00000000000000000000000000000000 P 688fafde05fa4ce0969f3d930d7b7b29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:56:09.252455 5452 raft_consensus.cc:482] T 00000000000000000000000000000000 P 688fafde05fa4ce0969f3d930d7b7b29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:56:09.252503 5452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 688fafde05fa4ce0969f3d930d7b7b29 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:56:09.253484 5452 raft_consensus.cc:504] T 00000000000000000000000000000000 P 688fafde05fa4ce0969f3d930d7b7b29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688fafde05fa4ce0969f3d930d7b7b29" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 40863 } }
I20260928 08:56:09.253602 5452 leader_election.cc:302] T 00000000000000000000000000000000 P 688fafde05fa4ce0969f3d930d7b7b29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 688fafde05fa4ce0969f3d930d7b7b29; no voters:
I20260928 08:56:09.253772 5452 leader_election.cc:288] T 00000000000000000000000000000000 P 688fafde05fa4ce0969f3d930d7b7b29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:56:09.253800 5457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 688fafde05fa4ce0969f3d930d7b7b29 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:56:09.253983 5457 raft_consensus.cc:686] T 00000000000000000000000000000000 P 688fafde05fa4ce0969f3d930d7b7b29 [term 1 LEADER]: Becoming Leader. State: Replica: 688fafde05fa4ce0969f3d930d7b7b29, State: Running, Role: LEADER
I20260928 08:56:09.254032 5452 sys_catalog.cc:564] T 00000000000000000000000000000000 P 688fafde05fa4ce0969f3d930d7b7b29 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:56:09.254084 5457 consensus_queue.cc:237] T 00000000000000000000000000000000 P 688fafde05fa4ce0969f3d930d7b7b29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688fafde05fa4ce0969f3d930d7b7b29" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 40863 } }
I20260928 08:56:09.254423 5457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 688fafde05fa4ce0969f3d930d7b7b29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 688fafde05fa4ce0969f3d930d7b7b29. Latest consensus state: current_term: 1 leader_uuid: "688fafde05fa4ce0969f3d930d7b7b29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688fafde05fa4ce0969f3d930d7b7b29" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 40863 } } }
I20260928 08:56:09.255260 5457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 688fafde05fa4ce0969f3d930d7b7b29 [sys.catalog]: This master's current role is: LEADER
I20260928 08:56:09.255172 5458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 688fafde05fa4ce0969f3d930d7b7b29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "688fafde05fa4ce0969f3d930d7b7b29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688fafde05fa4ce0969f3d930d7b7b29" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 40863 } } }
I20260928 08:56:09.255599 5458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 688fafde05fa4ce0969f3d930d7b7b29 [sys.catalog]: This master's current role is: LEADER
W20260928 08:56:09.255789 5471 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 688fafde05fa4ce0969f3d930d7b7b29: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:56:09.255832 5471 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:56:09.255887 5474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:56:09.256036 5474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:56:09.257571 5474 catalog_manager.cc:1347] Generated new cluster ID: 21fe7f161a564d28b354d26dd099fce7
I20260928 08:56:09.257661 5474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 08:56:09 dist-test-slave-pbjp krb5kdc[5371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:56:09.264371 5456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:09.252769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57314 (local address 127.23.167.254:40863)
0928 08:56:09.252937 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:09.252947 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:09.252982 (+ 35us) server_negotiation.cc:407] Connection header received
0928 08:56:09.253179 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:09.253185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:09.253230 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:09.253343 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:09.254131 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:09.255572 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:09.256596 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:09.257255 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:09.257359 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:09.260076 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:09.260080 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:09.260081 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:09.260093 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:09.262742 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:09.263124 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:09.263128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:09.263128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:09.263253 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:09.263425 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:09.263426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:09.263426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:09.263752 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:09.263904 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:09.263989 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 08:56:09.264146 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260928 08:56:09.265339 24223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:56:09.272305 5474 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:56:09.272886 5474 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:56:09.276160 5474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 688fafde05fa4ce0969f3d930d7b7b29: Generated new TSK 0
I20260928 08:56:09.276364 5474 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:56:09.319072 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 5384
2026-09-28T08:56:09Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1249 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:56:09 dist-test-slave-pbjp krb5kdc[5479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:56:09 dist-test-slave-pbjp krb5kdc[5479](info): set up 2 sockets
Sep 28 08:56:09 dist-test-slave-pbjp krb5kdc[5479](info): commencing operation
krb5kdc: starting...
W20260928 08:56:10.260985 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:56:10 dist-test-slave-pbjp krb5kdc[5479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:56:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:56:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.167.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
I20260928 08:56:10.373028 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:42107
--webserver_interface=127.23.167.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:42107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:56:10.451894 5492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:56:10.452093 5492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:56:10.452114 5492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:56:10.453238 5492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:56:10.453274 5492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:56:10.453285 5492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:56:10.453296 5492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:56:10.453310 5492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:56:10.454695 5492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:42107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:42107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:56:10.454890 5492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:56:10.455067 5492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:56:10.459802 5500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:10.459834 5497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:10.459848 5498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:10.460055 5492 server_base.cc:1034] running on GCE node
I20260928 08:56:10.460279 5492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:56:10.460563 5492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:56:10.461712 5492 hybrid_clock.cc:648] HybridClock initialized: now 1790585770461697 us; error 28 us; skew 500 ppm
Sep 28 08:56:10 dist-test-slave-pbjp krb5kdc[5479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585770, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:10.467619 5492 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:56:10.468060 5492 webserver.cc:460] Webserver started at http://127.23.167.254:44571/ using document root <none> and password file <none>
I20260928 08:56:10.468266 5492 fs_manager.cc:359] Metadata directory not provided
I20260928 08:56:10.468312 5492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:56:10.468441 5492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:56:10.469550 5492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "7c14df0884684549b1c32bec060cd0e0"
format_stamp: "Formatted at 2026-09-28 08:56:10 on dist-test-slave-pbjp"
server_key: "31eef92ae8608ff691b1846a873e5713"
server_key_iv: "b1e7423693030f1a3b589ff2e666d235"
server_key_version: "encryptionkey@0"
I20260928 08:56:10.469889 5492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "7c14df0884684549b1c32bec060cd0e0"
format_stamp: "Formatted at 2026-09-28 08:56:10 on dist-test-slave-pbjp"
server_key: "31eef92ae8608ff691b1846a873e5713"
server_key_iv: "b1e7423693030f1a3b589ff2e666d235"
server_key_version: "encryptionkey@0"
I20260928 08:56:10.471119 5492 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:56:10.471774 5507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:10.471928 5492 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:56:10.472010 5492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "7c14df0884684549b1c32bec060cd0e0"
format_stamp: "Formatted at 2026-09-28 08:56:10 on dist-test-slave-pbjp"
server_key: "31eef92ae8608ff691b1846a873e5713"
server_key_iv: "b1e7423693030f1a3b589ff2e666d235"
server_key_version: "encryptionkey@0"
I20260928 08:56:10.472070 5492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:56:10.484460 5492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:56:10.486264 5492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:56:10.486402 5492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:56:10.490271 5492 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:42107
I20260928 08:56:10.490302 5559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:42107 every 8 connection(s)
I20260928 08:56:10.490633 5492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:56:10.491560 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:56:10.493882 5560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0: Bootstrap starting.
I20260928 08:56:10.494400 5560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0: Neither blocks nor log segments found. Creating new log.
I20260928 08:56:10.494624 5560 log.cc:824] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0: Log is configured to *not* fsync() on all Append() calls
I20260928 08:56:10.495223 5560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0: No bootstrap required, opened a new log
I20260928 08:56:10.496752 5560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c14df0884684549b1c32bec060cd0e0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42107 } }
I20260928 08:56:10.496889 5560 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:56:10.496910 5560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c14df0884684549b1c32bec060cd0e0, State: Initialized, Role: FOLLOWER
I20260928 08:56:10.497025 5560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c14df0884684549b1c32bec060cd0e0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42107 } }
I20260928 08:56:10.497130 5560 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:56:10.497162 5560 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:56:10.497198 5560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:56:10.497277 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 5492
I20260928 08:56:10.497344 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:56:10.497483 24223 external_mini_cluster.cc:1428] Setting key 1bc4d300c24aa5dcbb9bae40ad147d39
I20260928 08:56:10.498013 5560 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c14df0884684549b1c32bec060cd0e0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42107 } }
I20260928 08:56:10.498128 5560 leader_election.cc:302] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c14df0884684549b1c32bec060cd0e0; no voters:
I20260928 08:56:10.498270 5560 leader_election.cc:288] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:56:10.498302 5565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:56:10.498463 5565 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [term 1 LEADER]: Becoming Leader. State: Replica: 7c14df0884684549b1c32bec060cd0e0, State: Running, Role: LEADER
I20260928 08:56:10.498513 5560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:56:10.498566 5565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c14df0884684549b1c32bec060cd0e0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42107 } }
I20260928 08:56:10.498955 5566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c14df0884684549b1c32bec060cd0e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c14df0884684549b1c32bec060cd0e0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42107 } } }
I20260928 08:56:10.499070 5565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c14df0884684549b1c32bec060cd0e0. Latest consensus state: current_term: 1 leader_uuid: "7c14df0884684549b1c32bec060cd0e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c14df0884684549b1c32bec060cd0e0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42107 } } }
I20260928 08:56:10.499133 5566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [sys.catalog]: This master's current role is: LEADER
I20260928 08:56:10.499150 5565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [sys.catalog]: This master's current role is: LEADER
I20260928 08:56:10.499905 5574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:56:10.500512 5574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:56:10.501998 5574 catalog_manager.cc:1347] Generated new cluster ID: f0867759b8b3417fb40da338becb8163
I20260928 08:56:10.502048 5574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 08:56:10 dist-test-slave-pbjp krb5kdc[5479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:56:10.508241 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:10.497925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47938 (local address 127.23.167.254:42107)
0928 08:56:10.498086 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:10.498095 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:10.498129 (+ 34us) server_negotiation.cc:407] Connection header received
0928 08:56:10.498246 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:10.498252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:10.498310 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:10.499222 (+ 912us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:10.499316 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:10.500534 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:10.501377 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:10.501951 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:10.502420 (+ 469us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:10.504756 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:10.504760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:10.504760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:10.504771 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:10.506656 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:10.507007 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:10.507012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:10.507012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:10.507159 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:10.507333 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:10.507334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:10.507334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:10.507652 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:10.507803 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:10.507882 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 08:56:10.508023 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
I20260928 08:56:10.510119 5574 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:56:10.510813 5574 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:56:10.522245 5574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0: Generated new TSK 0
I20260928 08:56:10.522439 5574 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.167.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
I20260928 08:56:10.547955 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:0
--local_ip_for_outbound_sockets=127.23.167.193
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:42107
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:56:10.628545 5586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:56:10.628789 5586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:56:10.628818 5586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:56:10.630129 5586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:56:10.630184 5586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:56:10.630214 5586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:56:10.631759 5586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:42107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:56:10.631994 5586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:56:10.632205 5586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:56:10.637013 5594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:10.637144 5586 server_base.cc:1034] running on GCE node
W20260928 08:56:10.637019 5592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:10.637032 5591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:10.637432 5586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:56:10.637653 5586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:56:10.638797 5586 hybrid_clock.cc:648] HybridClock initialized: now 1790585770638787 us; error 30 us; skew 500 ppm
Sep 28 08:56:10 dist-test-slave-pbjp krb5kdc[5479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585770, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:10.641183 5586 init.cc:377] Logged in from keytab as kudu/127.23.167.193@KRBTEST.COM (short username kudu)
I20260928 08:56:10.641561 5586 webserver.cc:460] Webserver started at http://127.23.167.193:41551/ using document root <none> and password file <none>
I20260928 08:56:10.641752 5586 fs_manager.cc:359] Metadata directory not provided
I20260928 08:56:10.641791 5586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:56:10.641878 5586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:56:10.642910 5586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/data/instance:
uuid: "6d135091233e4b609b1a7a67b7e5278f"
format_stamp: "Formatted at 2026-09-28 08:56:10 on dist-test-slave-pbjp"
server_key: "b8a4c345b7642c56e6507e056631274b"
server_key_iv: "07643772ecf3f758cbbe569f3a01d446"
server_key_version: "encryptionkey@0"
I20260928 08:56:10.643206 5586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance:
uuid: "6d135091233e4b609b1a7a67b7e5278f"
format_stamp: "Formatted at 2026-09-28 08:56:10 on dist-test-slave-pbjp"
server_key: "b8a4c345b7642c56e6507e056631274b"
server_key_iv: "07643772ecf3f758cbbe569f3a01d446"
server_key_version: "encryptionkey@0"
I20260928 08:56:10.644349 5586 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:56:10.645004 5601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:10.645170 5586 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:56:10.645264 5586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "6d135091233e4b609b1a7a67b7e5278f"
format_stamp: "Formatted at 2026-09-28 08:56:10 on dist-test-slave-pbjp"
server_key: "b8a4c345b7642c56e6507e056631274b"
server_key_iv: "07643772ecf3f758cbbe569f3a01d446"
server_key_version: "encryptionkey@0"
I20260928 08:56:10.645332 5586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:56:10.656615 5586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:56:10.658747 5586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:56:10.658885 5586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:56:10.659121 5586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:56:10.659430 5586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:56:10.659463 5586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:10.659494 5586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:56:10.659518 5586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:10.664915 5586 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:34943
I20260928 08:56:10.664976 5714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:34943 every 8 connection(s)
I20260928 08:56:10.665302 5586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
Sep 28 08:56:10 dist-test-slave-pbjp krb5kdc[5479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585770, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:56:10.672557 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 5586
I20260928 08:56:10.672655 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance
I20260928 08:56:10.672847 24223 external_mini_cluster.cc:1428] Setting key 928ee96f9d4e067ccc7a542f4c1b0d61
I20260928 08:56:10.674132 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:10.665991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:51595 (local address 127.23.167.254:42107)
0928 08:56:10.666074 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:10.666076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:10.666257 (+ 181us) server_negotiation.cc:407] Connection header received
0928 08:56:10.666740 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:10.666741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:10.666751 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:10.666802 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:10.667702 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:10.668215 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:10.669276 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:10.669527 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:10.669582 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:10.671544 (+ 1962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:10.671546 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:10.671546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:10.671553 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:10.672609 (+ 1056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:10.673126 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:10.673127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:10.673128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:10.673181 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:10.673489 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:10.673489 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:10.673490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:10.673595 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:10.673654 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:10.673939 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 08:56:10.674036 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 08:56:10.674273 5717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:10.665966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.193:51595)
0928 08:56:10.666133 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:10.666170 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:10.666659 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:10.666864 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:10.666867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:10.667424 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:56:10.667626 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:10.667632 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:10.668285 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:10.668285 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:10.669166 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:10.669169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:10.669595 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:10.669596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:10.669702 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:10.670138 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:56:10.670147 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:56:10.671454 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:56:10.672683 (+ 1229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:10.672691 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:10.672693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:10.672932 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:10.673237 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:10.673238 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:10.673239 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:10.673344 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:10.673654 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:56:10.673661 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:56:10.673796 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 08:56:10.673889 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 08:56:10.674046 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260928 08:56:10.674782 5715 heartbeater.cc:347] Connected to a master server at 127.23.167.254:42107
I20260928 08:56:10.674882 5715 heartbeater.cc:464] Registering TS with master...
I20260928 08:56:10.675072 5715 heartbeater.cc:511] Master 127.23.167.254:42107 requested a full tablet report, sending...
I20260928 08:56:10.675585 5524 ts_manager.cc:194] Registered new tserver with Master: 6d135091233e4b609b1a7a67b7e5278f (127.23.167.193:34943)
I20260928 08:56:10.676236 5524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.193@KRBTEST.COM'} at 127.23.167.193:51595
WARNING: no policy specified for kudu/127.23.167.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
I20260928 08:56:10.695643 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:0
--local_ip_for_outbound_sockets=127.23.167.194
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:42107
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:56:10.786850 5720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:56:10.787050 5720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:56:10.787081 5720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:56:10.788197 5720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:56:10.788236 5720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:56:10.788269 5720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:56:10.789726 5720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:42107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:56:10.789944 5720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:56:10.790144 5720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:56:10.794847 5728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:10.794885 5725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:10.794888 5726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:10.794940 5720 server_base.cc:1034] running on GCE node
I20260928 08:56:10.795226 5720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:56:10.795416 5720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:56:10.796543 5720 hybrid_clock.cc:648] HybridClock initialized: now 1790585770796531 us; error 29 us; skew 500 ppm
Sep 28 08:56:10 dist-test-slave-pbjp krb5kdc[5479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585770, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:10.798965 5720 init.cc:377] Logged in from keytab as kudu/127.23.167.194@KRBTEST.COM (short username kudu)
I20260928 08:56:10.799459 5720 webserver.cc:460] Webserver started at http://127.23.167.194:43755/ using document root <none> and password file <none>
I20260928 08:56:10.799666 5720 fs_manager.cc:359] Metadata directory not provided
I20260928 08:56:10.799712 5720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:56:10.799836 5720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:56:10.800894 5720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/data/instance:
uuid: "00f838a4f4cc497bacd133ba8a2999b3"
format_stamp: "Formatted at 2026-09-28 08:56:10 on dist-test-slave-pbjp"
server_key: "4543a003133dbcb799b400a74c1f57d4"
server_key_iv: "883c88a66dcec23959f58efae8b44b2f"
server_key_version: "encryptionkey@0"
I20260928 08:56:10.801241 5720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance:
uuid: "00f838a4f4cc497bacd133ba8a2999b3"
format_stamp: "Formatted at 2026-09-28 08:56:10 on dist-test-slave-pbjp"
server_key: "4543a003133dbcb799b400a74c1f57d4"
server_key_iv: "883c88a66dcec23959f58efae8b44b2f"
server_key_version: "encryptionkey@0"
I20260928 08:56:10.802378 5720 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:56:10.803007 5735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:10.803164 5720 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:56:10.803246 5720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "00f838a4f4cc497bacd133ba8a2999b3"
format_stamp: "Formatted at 2026-09-28 08:56:10 on dist-test-slave-pbjp"
server_key: "4543a003133dbcb799b400a74c1f57d4"
server_key_iv: "883c88a66dcec23959f58efae8b44b2f"
server_key_version: "encryptionkey@0"
I20260928 08:56:10.803308 5720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:56:10.819343 5720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:56:10.821580 5720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:56:10.821715 5720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:56:10.821926 5720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:56:10.822237 5720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:56:10.822270 5720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:10.822299 5720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:56:10.822324 5720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:10.828006 5720 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:34305
I20260928 08:56:10.828078 5848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:34305 every 8 connection(s)
I20260928 08:56:10.828364 5720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
I20260928 08:56:10.830119 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 5720
I20260928 08:56:10.830206 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance
I20260928 08:56:10.830399 24223 external_mini_cluster.cc:1428] Setting key 6f698a293917969db39e2a8d66357dfe
Sep 28 08:56:10 dist-test-slave-pbjp krb5kdc[5479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585770, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:56:10.838083 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:10.829127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:50283 (local address 127.23.167.254:42107)
0928 08:56:10.829202 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:10.829205 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:10.829370 (+ 165us) server_negotiation.cc:407] Connection header received
0928 08:56:10.829918 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:10.829920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:10.829932 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:10.830007 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:10.831094 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:10.831597 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:10.832590 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:10.832847 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:10.832923 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:10.835291 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:10.835293 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:10.835293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:10.835302 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:10.836730 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:10.837142 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:10.837143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:10.837144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:10.837208 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:10.837441 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:10.837442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:10.837442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:10.837587 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:10.837649 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:10.837899 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 08:56:10.837989 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 08:56:10.838140 5851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:10.829047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.194:50283)
0928 08:56:10.829245 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:10.829289 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:10.829807 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:10.830085 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:10.830090 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:10.830777 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:56:10.831004 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:10.831011 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:10.831672 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:10.831673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:10.832496 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:10.832498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:10.832978 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:10.832979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:10.833139 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:10.833641 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:56:10.833650 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:56:10.835188 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:56:10.836808 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:10.836815 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:10.836817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:10.837029 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:10.837273 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:10.837274 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:10.837274 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:10.837380 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:10.837640 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:56:10.837643 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:56:10.837759 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 08:56:10.837843 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 08:56:10.837963 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":80,"threads_started":1}
I20260928 08:56:10.838629 5849 heartbeater.cc:347] Connected to a master server at 127.23.167.254:42107
I20260928 08:56:10.838716 5849 heartbeater.cc:464] Registering TS with master...
I20260928 08:56:10.838881 5849 heartbeater.cc:511] Master 127.23.167.254:42107 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.167.195@KRBTEST.COM; defaulting to no policy
I20260928 08:56:10.839247 5524 ts_manager.cc:194] Registered new tserver with Master: 00f838a4f4cc497bacd133ba8a2999b3 (127.23.167.194:34305)
I20260928 08:56:10.839658 5524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.194@KRBTEST.COM'} at 127.23.167.194:50283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
I20260928 08:56:10.855929 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:0
--local_ip_for_outbound_sockets=127.23.167.195
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:42107
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:56:10.935042 5854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:56:10.935249 5854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:56:10.935271 5854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:56:10.936398 5854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:56:10.936436 5854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:56:10.936462 5854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:56:10.937901 5854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:42107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:56:10.938126 5854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:56:10.938331 5854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:56:10.943035 5860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:10.943025 5862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:10.943037 5859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:10.943070 5854 server_base.cc:1034] running on GCE node
I20260928 08:56:10.943416 5854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:56:10.943657 5854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:56:10.944803 5854 hybrid_clock.cc:648] HybridClock initialized: now 1790585770944788 us; error 31 us; skew 500 ppm
Sep 28 08:56:10 dist-test-slave-pbjp krb5kdc[5479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585770, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:10.947402 5854 init.cc:377] Logged in from keytab as kudu/127.23.167.195@KRBTEST.COM (short username kudu)
I20260928 08:56:10.947835 5854 webserver.cc:460] Webserver started at http://127.23.167.195:34753/ using document root <none> and password file <none>
I20260928 08:56:10.948063 5854 fs_manager.cc:359] Metadata directory not provided
I20260928 08:56:10.948114 5854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:56:10.948244 5854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:56:10.949378 5854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/data/instance:
uuid: "1f9e21caa9434b1f87809d96b2d26cfa"
format_stamp: "Formatted at 2026-09-28 08:56:10 on dist-test-slave-pbjp"
server_key: "e891a56e15f58bd3ca060536bc930ea7"
server_key_iv: "d17ace65586d2fb4a23592997540fc2a"
server_key_version: "encryptionkey@0"
I20260928 08:56:10.949713 5854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance:
uuid: "1f9e21caa9434b1f87809d96b2d26cfa"
format_stamp: "Formatted at 2026-09-28 08:56:10 on dist-test-slave-pbjp"
server_key: "e891a56e15f58bd3ca060536bc930ea7"
server_key_iv: "d17ace65586d2fb4a23592997540fc2a"
server_key_version: "encryptionkey@0"
I20260928 08:56:10.951021 5854 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:56:10.951684 5869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:10.951831 5854 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:56:10.951913 5854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "1f9e21caa9434b1f87809d96b2d26cfa"
format_stamp: "Formatted at 2026-09-28 08:56:10 on dist-test-slave-pbjp"
server_key: "e891a56e15f58bd3ca060536bc930ea7"
server_key_iv: "d17ace65586d2fb4a23592997540fc2a"
server_key_version: "encryptionkey@0"
I20260928 08:56:10.951984 5854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:56:10.962002 5854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:56:10.963958 5854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:56:10.964078 5854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:56:10.964259 5854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:56:10.964530 5854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:56:10.964562 5854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:10.964589 5854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:56:10.964612 5854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:10.969897 5854 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:34325
I20260928 08:56:10.969949 5982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:34325 every 8 connection(s)
I20260928 08:56:10.970240 5854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
Sep 28 08:56:10 dist-test-slave-pbjp krb5kdc[5479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585770, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:56:10.978637 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:10.970838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:46313 (local address 127.23.167.254:42107)
0928 08:56:10.970880 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:10.970882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:10.971111 (+ 229us) server_negotiation.cc:407] Connection header received
0928 08:56:10.971540 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:10.971541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:10.971553 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:10.971616 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:10.972517 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:10.973003 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:10.973986 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:10.974169 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:10.974215 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:10.976206 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:10.976207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:10.976208 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:10.976214 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:10.977433 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:10.977785 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:10.977786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:10.977787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:10.977837 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:10.978051 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:10.978052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:10.978052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:10.978159 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:10.978213 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:10.978465 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 08:56:10.978540 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 08:56:10.978786 5985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:10.970817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.195:46313)
0928 08:56:10.970964 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:10.971005 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:10.971466 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:10.971673 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:10.971693 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:10.972223 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:56:10.972404 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:10.972409 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:10.973099 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:10.973100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:10.973921 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:10.973923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:10.974249 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:10.974251 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:10.974384 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:10.974791 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:56:10.974798 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:56:10.976116 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:56:10.977504 (+ 1388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:10.977511 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:10.977513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:10.977721 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:10.977888 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:10.977889 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:10.977889 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:10.977996 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:10.978210 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:56:10.978216 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:56:10.978348 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 08:56:10.978432 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 08:56:10.978580 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
I20260928 08:56:10.979211 5983 heartbeater.cc:347] Connected to a master server at 127.23.167.254:42107
I20260928 08:56:10.979290 5983 heartbeater.cc:464] Registering TS with master...
I20260928 08:56:10.979463 5983 heartbeater.cc:511] Master 127.23.167.254:42107 requested a full tablet report, sending...
I20260928 08:56:10.979787 5524 ts_manager.cc:194] Registered new tserver with Master: 1f9e21caa9434b1f87809d96b2d26cfa (127.23.167.195:34325)
I20260928 08:56:10.980182 5524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.195@KRBTEST.COM'} at 127.23.167.195:46313
I20260928 08:56:10.980330 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 5854
I20260928 08:56:10.980430 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance
I20260928 08:56:10.980667 24223 external_mini_cluster.cc:1428] Setting key c2bb8f443fdfa1f9e02c2f1c96b9248d
I20260928 08:56:10.981411 24223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:56:10.987920 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:10.982663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47950 (local address 127.23.167.254:42107)
0928 08:56:10.982714 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:10.982716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:10.982795 (+ 79us) server_negotiation.cc:407] Connection header received
0928 08:56:10.982831 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:10.982832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:10.982839 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:10.982880 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:10.983353 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:10.983815 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:10.984481 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:10.984657 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:10.984696 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:10.985721 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:10.985722 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:10.985723 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:10.985728 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:10.986967 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:10.987275 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:10.987275 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:10.987275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:10.987329 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:10.987517 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:10.987517 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:10.987518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:10.987648 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:10.987715 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:10.987811 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 08:56:10.987861 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 08:56:10.989243 5524 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:56:10.990710 5524 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:56:10.996532 5992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:10.992669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:34325 (local address 127.0.0.1:45804)
0928 08:56:10.993105 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:10.993145 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:10.993212 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:10.993443 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:10.993444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:10.993446 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:56:10.993500 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:10.993501 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:10.994382 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:10.994383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:10.995456 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:10.995459 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:10.996258 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:10.996259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:10.996332 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:10.996345 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 08:56:10.996409 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 08:56:10.996457 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":57,"threads_started":1}
I20260928 08:56:10.996543 5994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:10.992748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45804 (local address 127.23.167.195:34325)
0928 08:56:10.992913 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:10.992918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:10.993211 (+ 293us) server_negotiation.cc:407] Connection header received
0928 08:56:10.993287 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:10.993289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:10.993305 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:10.993377 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:56:10.993551 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:10.994293 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:10.995545 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:10.996189 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:10.996249 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:10.996305 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:10.996428 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 08:56:10.996470 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260928 08:56:10.996973 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:10.992670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:34305 (local address 127.0.0.1:49654)
0928 08:56:10.993135 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:10.993149 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:10.993213 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:10.993384 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:10.993387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:10.993389 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:56:10.993462 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:10.993463 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:10.994822 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:10.994823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:10.995889 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:10.995891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:10.996746 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:10.996747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:10.996806 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:10.996810 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:56:10.996860 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 08:56:10.996897 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":1,"spinlock_wait_cycles":384,"thread_start_us":86,"threads_started":1}
I20260928 08:56:10.997153 5996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:10.992904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59470 (local address 127.23.167.193:34943)
0928 08:56:10.993086 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:10.993090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:10.993532 (+ 442us) server_negotiation.cc:407] Connection header received
0928 08:56:10.993579 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:10.993582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:10.993596 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:10.993666 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:56:10.993842 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:10.994856 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:10.995882 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:10.996601 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:10.996674 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:10.996732 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:10.997017 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 08:56:10.997056 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
I20260928 08:56:10.997221 5991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:10.992674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:34943 (local address 127.0.0.1:59470)
0928 08:56:10.993309 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:10.993317 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:10.993363 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:10.993738 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:10.993739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:10.993740 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:56:10.993773 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:10.993774 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:10.994938 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:10.994939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:10.995813 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:10.995816 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:10.996767 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:10.996768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:10.996830 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:10.996834 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:56:10.996888 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 08:56:10.996926 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":66,"threads_started":1}
I20260928 08:56:10.997534 5917 tablet_service.cc:1467] Processing CreateTablet for tablet 9fcdc12c61f34ed2a2c0084a8d89112f (DEFAULT_TABLE table=test-table [id=c90ffccf47c84e2ea88bb29cd05916a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:56:10.997867 5917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fcdc12c61f34ed2a2c0084a8d89112f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:56:10.997908 5649 tablet_service.cc:1467] Processing CreateTablet for tablet 9fcdc12c61f34ed2a2c0084a8d89112f (DEFAULT_TABLE table=test-table [id=c90ffccf47c84e2ea88bb29cd05916a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:56:10.998230 5649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fcdc12c61f34ed2a2c0084a8d89112f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:56:11.000629 5997 tablet_bootstrap.cc:492] T 9fcdc12c61f34ed2a2c0084a8d89112f P 1f9e21caa9434b1f87809d96b2d26cfa: Bootstrap starting.
I20260928 08:56:11.000787 5995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:10.992892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49654 (local address 127.23.167.194:34305)
0928 08:56:10.993083 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:10.993088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:10.993209 (+ 121us) server_negotiation.cc:407] Connection header received
0928 08:56:10.993290 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:10.993293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:10.993307 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:10.993380 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:56:10.993620 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:10.994721 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:10.995963 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:10.996674 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:10.996738 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:10.996782 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:10.996874 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 08:56:10.996913 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":76,"threads_started":1}
I20260928 08:56:11.000977 5998 tablet_bootstrap.cc:492] T 9fcdc12c61f34ed2a2c0084a8d89112f P 6d135091233e4b609b1a7a67b7e5278f: Bootstrap starting.
I20260928 08:56:11.001349 5783 tablet_service.cc:1467] Processing CreateTablet for tablet 9fcdc12c61f34ed2a2c0084a8d89112f (DEFAULT_TABLE table=test-table [id=c90ffccf47c84e2ea88bb29cd05916a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:56:11.001492 5997 tablet_bootstrap.cc:654] T 9fcdc12c61f34ed2a2c0084a8d89112f P 1f9e21caa9434b1f87809d96b2d26cfa: Neither blocks nor log segments found. Creating new log.
I20260928 08:56:11.001622 5783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fcdc12c61f34ed2a2c0084a8d89112f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:56:11.001623 5998 tablet_bootstrap.cc:654] T 9fcdc12c61f34ed2a2c0084a8d89112f P 6d135091233e4b609b1a7a67b7e5278f: Neither blocks nor log segments found. Creating new log.
I20260928 08:56:11.002133 5997 log.cc:824] T 9fcdc12c61f34ed2a2c0084a8d89112f P 1f9e21caa9434b1f87809d96b2d26cfa: Log is configured to *not* fsync() on all Append() calls
I20260928 08:56:11.002134 5998 log.cc:824] T 9fcdc12c61f34ed2a2c0084a8d89112f P 6d135091233e4b609b1a7a67b7e5278f: Log is configured to *not* fsync() on all Append() calls
I20260928 08:56:11.002817 5998 tablet_bootstrap.cc:492] T 9fcdc12c61f34ed2a2c0084a8d89112f P 6d135091233e4b609b1a7a67b7e5278f: No bootstrap required, opened a new log
I20260928 08:56:11.002828 5997 tablet_bootstrap.cc:492] T 9fcdc12c61f34ed2a2c0084a8d89112f P 1f9e21caa9434b1f87809d96b2d26cfa: No bootstrap required, opened a new log
I20260928 08:56:11.002887 5997 ts_tablet_manager.cc:1397] T 9fcdc12c61f34ed2a2c0084a8d89112f P 1f9e21caa9434b1f87809d96b2d26cfa: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 08:56:11.002887 5998 ts_tablet_manager.cc:1397] T 9fcdc12c61f34ed2a2c0084a8d89112f P 6d135091233e4b609b1a7a67b7e5278f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:56:11.004125 6001 tablet_bootstrap.cc:492] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3: Bootstrap starting.
I20260928 08:56:11.004840 6001 tablet_bootstrap.cc:654] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3: Neither blocks nor log segments found. Creating new log.
I20260928 08:56:11.004748 5998 raft_consensus.cc:348] T 9fcdc12c61f34ed2a2c0084a8d89112f P 6d135091233e4b609b1a7a67b7e5278f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 } } peers { permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 } } peers { permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 } }
I20260928 08:56:11.004748 5997 raft_consensus.cc:348] T 9fcdc12c61f34ed2a2c0084a8d89112f P 1f9e21caa9434b1f87809d96b2d26cfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 } } peers { permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 } } peers { permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 } }
I20260928 08:56:11.004904 5998 raft_consensus.cc:374] T 9fcdc12c61f34ed2a2c0084a8d89112f P 6d135091233e4b609b1a7a67b7e5278f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:56:11.004904 5997 raft_consensus.cc:374] T 9fcdc12c61f34ed2a2c0084a8d89112f P 1f9e21caa9434b1f87809d96b2d26cfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:56:11.004946 5998 raft_consensus.cc:729] T 9fcdc12c61f34ed2a2c0084a8d89112f P 6d135091233e4b609b1a7a67b7e5278f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d135091233e4b609b1a7a67b7e5278f, State: Initialized, Role: FOLLOWER
I20260928 08:56:11.004959 5997 raft_consensus.cc:729] T 9fcdc12c61f34ed2a2c0084a8d89112f P 1f9e21caa9434b1f87809d96b2d26cfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9e21caa9434b1f87809d96b2d26cfa, State: Initialized, Role: FOLLOWER
I20260928 08:56:11.005080 5998 consensus_queue.cc:260] T 9fcdc12c61f34ed2a2c0084a8d89112f P 6d135091233e4b609b1a7a67b7e5278f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 } } peers { permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 } } peers { permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 } }
I20260928 08:56:11.005091 5997 consensus_queue.cc:260] T 9fcdc12c61f34ed2a2c0084a8d89112f P 1f9e21caa9434b1f87809d96b2d26cfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 } } peers { permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 } } peers { permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 } }
I20260928 08:56:11.005354 5997 ts_tablet_manager.cc:1428] T 9fcdc12c61f34ed2a2c0084a8d89112f P 1f9e21caa9434b1f87809d96b2d26cfa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:56:11.005354 5998 ts_tablet_manager.cc:1428] T 9fcdc12c61f34ed2a2c0084a8d89112f P 6d135091233e4b609b1a7a67b7e5278f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 08:56:11.005393 5715 heartbeater.cc:503] Master 127.23.167.254:42107 was elected leader, sending a full tablet report...
I20260928 08:56:11.005475 5983 heartbeater.cc:503] Master 127.23.167.254:42107 was elected leader, sending a full tablet report...
I20260928 08:56:11.005676 6001 log.cc:824] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3: Log is configured to *not* fsync() on all Append() calls
I20260928 08:56:11.006775 6001 tablet_bootstrap.cc:492] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3: No bootstrap required, opened a new log
I20260928 08:56:11.006827 6001 ts_tablet_manager.cc:1397] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 08:56:11.008133 6001 raft_consensus.cc:348] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 } } peers { permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 } } peers { permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 } }
I20260928 08:56:11.008260 6001 raft_consensus.cc:374] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:56:11.008291 6001 raft_consensus.cc:729] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00f838a4f4cc497bacd133ba8a2999b3, State: Initialized, Role: FOLLOWER
I20260928 08:56:11.008415 6001 consensus_queue.cc:260] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 } } peers { permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 } } peers { permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 } }
I20260928 08:56:11.008666 5849 heartbeater.cc:503] Master 127.23.167.254:42107 was elected leader, sending a full tablet report...
I20260928 08:56:11.008674 6001 ts_tablet_manager.cc:1428] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260928 08:56:11.078904 5850 tablet.cc:2369] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:56:11.166040 5716 tablet.cc:2369] T 9fcdc12c61f34ed2a2c0084a8d89112f P 6d135091233e4b609b1a7a67b7e5278f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:56:11.220764 5984 tablet.cc:2369] T 9fcdc12c61f34ed2a2c0084a8d89112f P 1f9e21caa9434b1f87809d96b2d26cfa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:56:11.238238 6005 raft_consensus.cc:482] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:56:11.238382 6005 raft_consensus.cc:504] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 } } peers { permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 } } peers { permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 } }
I20260928 08:56:11.238721 6005 leader_election.cc:288] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f9e21caa9434b1f87809d96b2d26cfa (127.23.167.195:34325), 6d135091233e4b609b1a7a67b7e5278f (127.23.167.193:34943)
I20260928 08:56:11.241503 5996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:11.238992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:47655 (local address 127.23.167.193:34943)
0928 08:56:11.239033 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:11.239035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:11.239139 (+ 104us) server_negotiation.cc:407] Connection header received
0928 08:56:11.239196 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:11.239196 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:11.239208 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:11.239262 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:56:11.239412 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.239895 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.240812 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.241197 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.241249 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:11.241258 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:11.241392 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 08:56:11.241429 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 08:56:11.241520 5994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:11.238980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:45905 (local address 127.23.167.195:34325)
0928 08:56:11.239081 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:11.239083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:11.239093 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:56:11.239124 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:11.239125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:11.239134 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:11.239189 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:56:11.239347 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.239875 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.240813 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.241218 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.241285 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:11.241294 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:11.241424 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 08:56:11.241461 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 08:56:11.241526 6006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:11.238921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:34943 (local address 127.23.167.194:47655)
0928 08:56:11.239058 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:11.239068 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:11.239142 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:11.239273 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:11.239274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:11.239275 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:56:11.239342 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:11.239343 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.239977 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.239977 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:11.240729 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:11.240731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.241263 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.241264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:11.241331 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:11.241335 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:56:11.241375 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 08:56:11.241409 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":2,"thread_start_us":91,"threads_started":1}
I20260928 08:56:11.241536 5851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:11.238891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:34325 (local address 127.23.167.194:45905)
0928 08:56:11.238977 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:11.238989 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:11.239033 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:11.239196 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:11.239196 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:11.239198 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:56:11.239272 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:11.239274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.239955 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.239956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:11.240718 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:11.240721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.241291 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.241292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:11.241355 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:11.241357 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 08:56:11.241408 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 08:56:11.241442 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":1}
I20260928 08:56:11.241866 5669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fcdc12c61f34ed2a2c0084a8d89112f" candidate_uuid: "00f838a4f4cc497bacd133ba8a2999b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d135091233e4b609b1a7a67b7e5278f" is_pre_election: true
I20260928 08:56:11.242029 5669 raft_consensus.cc:2454] T 9fcdc12c61f34ed2a2c0084a8d89112f P 6d135091233e4b609b1a7a67b7e5278f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00f838a4f4cc497bacd133ba8a2999b3 in term 0.
I20260928 08:56:11.242043 5937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fcdc12c61f34ed2a2c0084a8d89112f" candidate_uuid: "00f838a4f4cc497bacd133ba8a2999b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" is_pre_election: true
I20260928 08:56:11.242202 5937 raft_consensus.cc:2454] T 9fcdc12c61f34ed2a2c0084a8d89112f P 1f9e21caa9434b1f87809d96b2d26cfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00f838a4f4cc497bacd133ba8a2999b3 in term 0.
I20260928 08:56:11.242273 5739 leader_election.cc:302] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00f838a4f4cc497bacd133ba8a2999b3, 6d135091233e4b609b1a7a67b7e5278f; no voters:
I20260928 08:56:11.242393 6005 raft_consensus.cc:2781] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:56:11.242451 6005 raft_consensus.cc:482] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:56:11.242473 6005 raft_consensus.cc:3037] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:56:11.243186 6005 raft_consensus.cc:504] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 } } peers { permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 } } peers { permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 } }
I20260928 08:56:11.243326 6005 leader_election.cc:288] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3 [CANDIDATE]: Term 1 election: Requested vote from peers 1f9e21caa9434b1f87809d96b2d26cfa (127.23.167.195:34325), 6d135091233e4b609b1a7a67b7e5278f (127.23.167.193:34943)
I20260928 08:56:11.243446 5937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fcdc12c61f34ed2a2c0084a8d89112f" candidate_uuid: "00f838a4f4cc497bacd133ba8a2999b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9e21caa9434b1f87809d96b2d26cfa"
I20260928 08:56:11.243515 5937 raft_consensus.cc:3037] T 9fcdc12c61f34ed2a2c0084a8d89112f P 1f9e21caa9434b1f87809d96b2d26cfa [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:56:11.243511 5669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fcdc12c61f34ed2a2c0084a8d89112f" candidate_uuid: "00f838a4f4cc497bacd133ba8a2999b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d135091233e4b609b1a7a67b7e5278f"
I20260928 08:56:11.243587 5669 raft_consensus.cc:3037] T 9fcdc12c61f34ed2a2c0084a8d89112f P 6d135091233e4b609b1a7a67b7e5278f [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:56:11.244199 5937 raft_consensus.cc:2454] T 9fcdc12c61f34ed2a2c0084a8d89112f P 1f9e21caa9434b1f87809d96b2d26cfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00f838a4f4cc497bacd133ba8a2999b3 in term 1.
I20260928 08:56:11.244345 5669 raft_consensus.cc:2454] T 9fcdc12c61f34ed2a2c0084a8d89112f P 6d135091233e4b609b1a7a67b7e5278f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00f838a4f4cc497bacd133ba8a2999b3 in term 1.
I20260928 08:56:11.244374 5736 leader_election.cc:302] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00f838a4f4cc497bacd133ba8a2999b3, 1f9e21caa9434b1f87809d96b2d26cfa; no voters:
I20260928 08:56:11.244438 6005 raft_consensus.cc:2781] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:56:11.244573 6005 raft_consensus.cc:686] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3 [term 1 LEADER]: Becoming Leader. State: Replica: 00f838a4f4cc497bacd133ba8a2999b3, State: Running, Role: LEADER
I20260928 08:56:11.244668 6005 consensus_queue.cc:237] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 } } peers { permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 } } peers { permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 } }
I20260928 08:56:11.245460 5524 catalog_manager.cc:5495] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00f838a4f4cc497bacd133ba8a2999b3 (127.23.167.194). New cstate: current_term: 1 leader_uuid: "00f838a4f4cc497bacd133ba8a2999b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 } health_report { overall_health: UNKNOWN } } }
I20260928 08:56:11.278043 5995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:11.275729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49666 (local address 127.23.167.194:34305)
0928 08:56:11.275798 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:11.275800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:11.275808 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:56:11.275845 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:11.275846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:11.275867 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:11.275930 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:56:11.276052 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.276529 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.277308 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.277494 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.277550 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:11.277705 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:56:11.277812 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:56:11.277884 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:11.277921 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 08:56:11.277942 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 08:56:11.279839 5937 raft_consensus.cc:1261] T 9fcdc12c61f34ed2a2c0084a8d89112f P 1f9e21caa9434b1f87809d96b2d26cfa [term 1 FOLLOWER]: Refusing update from remote peer 00f838a4f4cc497bacd133ba8a2999b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:56:11.279838 5669 raft_consensus.cc:1261] T 9fcdc12c61f34ed2a2c0084a8d89112f P 6d135091233e4b609b1a7a67b7e5278f [term 1 FOLLOWER]: Refusing update from remote peer 00f838a4f4cc497bacd133ba8a2999b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:56:11.280176 6005 consensus_queue.cc:1035] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:56:11.280315 6007 consensus_queue.cc:1035] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:56:11.284833 5523 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:56:11.284916 5523 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:56:11.285671 5523 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0: Sending DeleteTablet for 3 replicas of tablet 9fcdc12c61f34ed2a2c0084a8d89112f
I20260928 08:56:11.285969 5649 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fcdc12c61f34ed2a2c0084a8d89112f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:56:11 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:59470
I20260928 08:56:11.286003 5783 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fcdc12c61f34ed2a2c0084a8d89112f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:56:11 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:49654
I20260928 08:56:11.286221 6021 tablet_replica.cc:330] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3: stopping tablet replica
I20260928 08:56:11.286233 5917 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fcdc12c61f34ed2a2c0084a8d89112f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:56:11 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:45804
I20260928 08:56:11.286331 6021 raft_consensus.cc:2229] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:56:11.286429 6022 tablet_replica.cc:330] T 9fcdc12c61f34ed2a2c0084a8d89112f P 1f9e21caa9434b1f87809d96b2d26cfa: stopping tablet replica
I20260928 08:56:11.286468 6021 raft_consensus.cc:2258] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:56:11.286496 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 5586
I20260928 08:56:11.286544 6022 raft_consensus.cc:2229] T 9fcdc12c61f34ed2a2c0084a8d89112f P 1f9e21caa9434b1f87809d96b2d26cfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:56:11.287204 6022 raft_consensus.cc:2258] T 9fcdc12c61f34ed2a2c0084a8d89112f P 1f9e21caa9434b1f87809d96b2d26cfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:56:11.287621 6022 ts_tablet_manager.cc:1905] T 9fcdc12c61f34ed2a2c0084a8d89112f P 1f9e21caa9434b1f87809d96b2d26cfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:56:11.289332 6022 ts_tablet_manager.cc:1918] T 9fcdc12c61f34ed2a2c0084a8d89112f P 1f9e21caa9434b1f87809d96b2d26cfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:56:11.289403 6022 log.cc:1196] T 9fcdc12c61f34ed2a2c0084a8d89112f P 1f9e21caa9434b1f87809d96b2d26cfa: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/wal/wals/9fcdc12c61f34ed2a2c0084a8d89112f
I20260928 08:56:11.289657 6022 ts_tablet_manager.cc:1939] T 9fcdc12c61f34ed2a2c0084a8d89112f P 1f9e21caa9434b1f87809d96b2d26cfa: Deleting consensus metadata
I20260928 08:56:11.290066 5508 catalog_manager.cc:4841] TS 1f9e21caa9434b1f87809d96b2d26cfa (127.23.167.195:34325): tablet 9fcdc12c61f34ed2a2c0084a8d89112f (table test-table [id=c90ffccf47c84e2ea88bb29cd05916a1]) successfully deleted
W20260928 08:56:11.291383 5511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.167.193:34943 (error 108)
W20260928 08:56:11.292615 5991 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:56:11.291748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:34943 (local address 127.0.0.1:59486)
0928 08:56:11.291799 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:11.291812 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:11.291859 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:11.292565 (+ 706us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.193:34943: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 08:56:11.292745 5511 catalog_manager.cc:4584] TS 6d135091233e4b609b1a7a67b7e5278f (127.23.167.193:34943): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9fcdc12c61f34ed2a2c0084a8d89112f: Network error: Client connection negotiation failed: client connection to 127.23.167.193:34943: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 08:56:11.292761 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 5720
I20260928 08:56:11.297979 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 5854
I20260928 08:56:11.298082 5991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:11.297955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:34305 (local address 127.0.0.1:49674)
0928 08:56:11.297999 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:11.298049 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.194:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 08:56:11.303184 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 5492
I20260928 08:56:11.308151 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:42107
--webserver_interface=127.23.167.254
--webserver_port=44571
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:42107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:56:11.389818 6024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:56:11.390017 6024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:56:11.390045 6024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:56:11.391155 6024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:56:11.391193 6024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:56:11.391211 6024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:56:11.391227 6024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:56:11.391242 6024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:56:11.392676 6024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:42107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:42107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=44571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:56:11.392915 6024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:56:11.393137 6024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:56:11.401054 6033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:11.401100 6031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:11.401046 6030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:11.401311 6024 server_base.cc:1034] running on GCE node
I20260928 08:56:11.401563 6024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:56:11.401839 6024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:56:11.403005 6024 hybrid_clock.cc:648] HybridClock initialized: now 1790585771402988 us; error 30 us; skew 500 ppm
Sep 28 08:56:11 dist-test-slave-pbjp krb5kdc[5479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585771, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:11.405336 6024 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:56:11.405750 6024 webserver.cc:460] Webserver started at http://127.23.167.254:44571/ using document root <none> and password file <none>
I20260928 08:56:11.405949 6024 fs_manager.cc:359] Metadata directory not provided
I20260928 08:56:11.405998 6024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:56:11.407344 6024 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:56:11.407948 6040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:11.408094 6024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:56:11.408182 6024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "7c14df0884684549b1c32bec060cd0e0"
format_stamp: "Formatted at 2026-09-28 08:56:10 on dist-test-slave-pbjp"
server_key: "31eef92ae8608ff691b1846a873e5713"
server_key_iv: "b1e7423693030f1a3b589ff2e666d235"
server_key_version: "encryptionkey@0"
I20260928 08:56:11.408598 6024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:56:11.419387 6024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:56:11.421155 6024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:56:11.421267 6024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:56:11.424899 6024 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:42107
I20260928 08:56:11.424942 6092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:42107 every 8 connection(s)
I20260928 08:56:11.425307 6024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:56:11.426838 6093 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:56:11.427453 6093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0: Bootstrap starting.
I20260928 08:56:11.432494 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 6024
I20260928 08:56:11.432744 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:34943
--local_ip_for_outbound_sockets=127.23.167.193
--tserver_master_addrs=127.23.167.254:42107
--webserver_port=41551
--webserver_interface=127.23.167.193
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:56:11.435281 6093 log.cc:824] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0: Log is configured to *not* fsync() on all Append() calls
I20260928 08:56:11.437244 6093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:56:11.437520 6093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0: Bootstrap complete.
I20260928 08:56:11.439644 6093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c14df0884684549b1c32bec060cd0e0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42107 } }
I20260928 08:56:11.439881 6093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c14df0884684549b1c32bec060cd0e0, State: Initialized, Role: FOLLOWER
I20260928 08:56:11.440032 6093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c14df0884684549b1c32bec060cd0e0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42107 } }
I20260928 08:56:11.440106 6093 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:56:11.440160 6093 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:56:11.440207 6093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [term 1 FOLLOWER]: Advancing to term 2
I20260928 08:56:11.441250 6093 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c14df0884684549b1c32bec060cd0e0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42107 } }
I20260928 08:56:11.441344 6093 leader_election.cc:302] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c14df0884684549b1c32bec060cd0e0; no voters:
I20260928 08:56:11.441506 6093 leader_election.cc:288] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 08:56:11.441556 6097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 08:56:11.441697 6097 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [term 2 LEADER]: Becoming Leader. State: Replica: 7c14df0884684549b1c32bec060cd0e0, State: Running, Role: LEADER
I20260928 08:56:11.441768 6093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:56:11.441848 6097 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c14df0884684549b1c32bec060cd0e0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42107 } }
I20260928 08:56:11.442122 6097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c14df0884684549b1c32bec060cd0e0. Latest consensus state: current_term: 2 leader_uuid: "7c14df0884684549b1c32bec060cd0e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c14df0884684549b1c32bec060cd0e0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42107 } } }
I20260928 08:56:11.442195 6097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [sys.catalog]: This master's current role is: LEADER
I20260928 08:56:11.442143 6098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7c14df0884684549b1c32bec060cd0e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c14df0884684549b1c32bec060cd0e0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42107 } } }
I20260928 08:56:11.442349 6098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [sys.catalog]: This master's current role is: LEADER
I20260928 08:56:11.443357 6113 catalog_manager.cc:1259] Loaded cluster ID: f0867759b8b3417fb40da338becb8163
I20260928 08:56:11.443396 6113 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0: loading cluster ID for follower catalog manager: success
W20260928 08:56:11.443580 6113 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260928 08:56:11.443639 6113 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260928 08:56:11.443771 6104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:56:11.444101 6104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:56:11.444177 6104 catalog_manager.cc:1259] Loaded cluster ID: f0867759b8b3417fb40da338becb8163
I20260928 08:56:11.444232 6104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:56:11.444391 6104 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 08:56:11.447134 6104 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 08:56:11.448932 6104 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:56:11.449185 6104 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0: Loaded TSK: 0
I20260928 08:56:11.449389 6104 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 08:56:11.544586 6095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:56:11.544894 6095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:56:11.544940 6095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:56:11.546700 6095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:56:11.546770 6095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:56:11.546813 6095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:56:11.549111 6095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:34943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=41551
--tserver_master_addrs=127.23.167.254:42107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:56:11.549396 6095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:56:11.549696 6095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:56:11.555207 6120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:11.555227 6121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:11.555207 6123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:11.555428 6095 server_base.cc:1034] running on GCE node
I20260928 08:56:11.555629 6095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:56:11.555898 6095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:56:11.557107 6095 hybrid_clock.cc:648] HybridClock initialized: now 1790585771557089 us; error 32 us; skew 500 ppm
Sep 28 08:56:11 dist-test-slave-pbjp krb5kdc[5479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585771, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:11.560595 6095 init.cc:377] Logged in from keytab as kudu/127.23.167.193@KRBTEST.COM (short username kudu)
I20260928 08:56:11.561014 6095 webserver.cc:460] Webserver started at http://127.23.167.193:41551/ using document root <none> and password file <none>
I20260928 08:56:11.561239 6095 fs_manager.cc:359] Metadata directory not provided
I20260928 08:56:11.561299 6095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:56:11.562655 6095 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:56:11.563321 6130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:11.563469 6095 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:56:11.563547 6095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "6d135091233e4b609b1a7a67b7e5278f"
format_stamp: "Formatted at 2026-09-28 08:56:10 on dist-test-slave-pbjp"
server_key: "b8a4c345b7642c56e6507e056631274b"
server_key_iv: "07643772ecf3f758cbbe569f3a01d446"
server_key_version: "encryptionkey@0"
I20260928 08:56:11.563896 6095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:56:11.586931 6095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:56:11.588999 6095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:56:11.589160 6095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:56:11.589378 6095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:56:11.589783 6137 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 08:56:11.591588 6095 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 08:56:11.591634 6095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 08:56:11.591665 6095 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 08:56:11.592263 6095 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 08:56:11.592293 6095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 08:56:11.592371 6137 tablet_bootstrap.cc:492] T 9fcdc12c61f34ed2a2c0084a8d89112f P 6d135091233e4b609b1a7a67b7e5278f: Bootstrap starting.
I20260928 08:56:11.598941 6095 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:34943
I20260928 08:56:11.599290 6095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
I20260928 08:56:11.602977 6137 log.cc:824] T 9fcdc12c61f34ed2a2c0084a8d89112f P 6d135091233e4b609b1a7a67b7e5278f: Log is configured to *not* fsync() on all Append() calls
I20260928 08:56:11.603233 6244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:34943 every 8 connection(s)
I20260928 08:56:11.605607 6137 tablet_bootstrap.cc:492] T 9fcdc12c61f34ed2a2c0084a8d89112f P 6d135091233e4b609b1a7a67b7e5278f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:56:11.605898 6137 tablet_bootstrap.cc:492] T 9fcdc12c61f34ed2a2c0084a8d89112f P 6d135091233e4b609b1a7a67b7e5278f: Bootstrap complete.
I20260928 08:56:11.607079 6137 ts_tablet_manager.cc:1397] T 9fcdc12c61f34ed2a2c0084a8d89112f P 6d135091233e4b609b1a7a67b7e5278f: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.009s
I20260928 08:56:11.607822 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 6095
I20260928 08:56:11.608088 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:34305
--local_ip_for_outbound_sockets=127.23.167.194
--tserver_master_addrs=127.23.167.254:42107
--webserver_port=43755
--webserver_interface=127.23.167.194
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:56:11.608790 6137 raft_consensus.cc:348] T 9fcdc12c61f34ed2a2c0084a8d89112f P 6d135091233e4b609b1a7a67b7e5278f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 } } peers { permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 } } peers { permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 } }
I20260928 08:56:11.609126 6137 raft_consensus.cc:729] T 9fcdc12c61f34ed2a2c0084a8d89112f P 6d135091233e4b609b1a7a67b7e5278f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d135091233e4b609b1a7a67b7e5278f, State: Initialized, Role: FOLLOWER
I20260928 08:56:11.609279 6137 consensus_queue.cc:260] T 9fcdc12c61f34ed2a2c0084a8d89112f P 6d135091233e4b609b1a7a67b7e5278f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 } } peers { permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 } } peers { permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 } }
I20260928 08:56:11.609535 6137 ts_tablet_manager.cc:1428] T 9fcdc12c61f34ed2a2c0084a8d89112f P 6d135091233e4b609b1a7a67b7e5278f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
Sep 28 08:56:11 dist-test-slave-pbjp krb5kdc[5479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585771, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:56:11.614040 6247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:11.599944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.193:40473)
0928 08:56:11.603392 (+ 3448us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:11.603438 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:11.603936 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:11.604470 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:11.604475 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:11.604768 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:56:11.604964 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:11.604970 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.606150 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.606152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:11.606969 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:11.606972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.607620 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.607622 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:11.607763 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:11.608355 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:56:11.608365 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:56:11.610138 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:56:11.612324 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:11.612331 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:11.612333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:11.612631 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:11.612920 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:11.612921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:11.612921 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:11.613017 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:11.613513 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:56:11.613518 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:56:11.613662 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 08:56:11.613747 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 08:56:11.613883 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3404,"thread_start_us":94,"threads_started":1}
I20260928 08:56:11.614194 6248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:11.600018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:40473 (local address 127.23.167.254:42107)
0928 08:56:11.604101 (+ 4083us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:11.604119 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:11.604150 (+ 31us) server_negotiation.cc:407] Connection header received
0928 08:56:11.604258 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:11.604262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:11.604306 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:11.604419 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:11.605124 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.606060 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.607053 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.607508 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.610153 (+ 2645us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:11.610263 (+ 110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:11.610266 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:11.610267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:11.610277 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:11.612160 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:11.612718 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:11.612722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:11.612722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:11.612854 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:11.613109 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:11.613109 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:11.613110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:11.613440 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:11.613528 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:11.613829 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 08:56:11.613982 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4037,"thread_start_us":67,"threads_started":1}
I20260928 08:56:11.614923 6245 heartbeater.cc:347] Connected to a master server at 127.23.167.254:42107
I20260928 08:56:11.615032 6245 heartbeater.cc:464] Registering TS with master...
I20260928 08:56:11.615235 6245 heartbeater.cc:511] Master 127.23.167.254:42107 requested a full tablet report, sending...
I20260928 08:56:11.616035 6057 ts_manager.cc:194] Registered new tserver with Master: 6d135091233e4b609b1a7a67b7e5278f (127.23.167.193:34943)
I20260928 08:56:11.617173 6057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.193@KRBTEST.COM'} at 127.23.167.193:40473
I20260928 08:56:11.618057 6245 heartbeater.cc:503] Master 127.23.167.254:42107 was elected leader, sending a full tablet report...
Sep 28 08:56:11 dist-test-slave-pbjp krb5kdc[5479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585771, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for kudu/127.23.167.193@KRBTEST.COM
I20260928 08:56:11.624874 6252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:11.616617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:34943 (local address 127.0.0.1:59490)
0928 08:56:11.616794 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:11.616836 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:11.616903 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:11.617196 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:11.617201 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:11.617516 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:56:11.617580 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:11.617583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.618857 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.618858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:11.619446 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:11.619448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.620045 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.620045 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:11.620101 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:11.620577 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:56:11.620582 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:56:11.622134 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:56:11.623847 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:11.623853 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:11.623855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:11.624037 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:11.624226 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:11.624227 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:11.624227 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:11.624278 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:11.624595 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:56:11.624601 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:56:11.624644 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 08:56:11.624709 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 08:56:11.624798 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":101,"threads_started":1}
I20260928 08:56:11.624943 6253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:11.616830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59490 (local address 127.23.167.193:34943)
0928 08:56:11.616973 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:11.616978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:11.616987 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:56:11.617024 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:11.617025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:11.617038 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:11.617132 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:11.617665 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.618777 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.619518 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.619981 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.620049 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:11.622212 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:11.622220 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:11.622221 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:11.622229 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:11.623773 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:11.624103 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:11.624105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:11.624106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:11.624171 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:11.624326 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:11.624327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:11.624328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:11.624511 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:11.624583 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:11.624752 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 08:56:11.624844 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
I20260928 08:56:11.625325 6179 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fcdc12c61f34ed2a2c0084a8d89112f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:56:11 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:59490
I20260928 08:56:11.625814 6254 tablet_replica.cc:330] stopping tablet replica
I20260928 08:56:11.625905 6254 raft_consensus.cc:2229] T 9fcdc12c61f34ed2a2c0084a8d89112f P 6d135091233e4b609b1a7a67b7e5278f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:56:11.625972 6254 raft_consensus.cc:2258] T 9fcdc12c61f34ed2a2c0084a8d89112f P 6d135091233e4b609b1a7a67b7e5278f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:56:11.626251 6254 ts_tablet_manager.cc:1905] T 9fcdc12c61f34ed2a2c0084a8d89112f P 6d135091233e4b609b1a7a67b7e5278f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:56:11.628242 6254 ts_tablet_manager.cc:1918] T 9fcdc12c61f34ed2a2c0084a8d89112f P 6d135091233e4b609b1a7a67b7e5278f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:56:11.628300 6254 log.cc:1196] T 9fcdc12c61f34ed2a2c0084a8d89112f P 6d135091233e4b609b1a7a67b7e5278f: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/wal/wals/9fcdc12c61f34ed2a2c0084a8d89112f
I20260928 08:56:11.628477 6254 ts_tablet_manager.cc:1939] T 9fcdc12c61f34ed2a2c0084a8d89112f P 6d135091233e4b609b1a7a67b7e5278f: Deleting consensus metadata
I20260928 08:56:11.628914 6044 catalog_manager.cc:4841] TS 6d135091233e4b609b1a7a67b7e5278f (127.23.167.193:34943): tablet 9fcdc12c61f34ed2a2c0084a8d89112f (table test-table [id=c90ffccf47c84e2ea88bb29cd05916a1]) successfully deleted
W20260928 08:56:11.693033 6250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:56:11.693300 6250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:56:11.693332 6250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:56:11.694487 6250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:56:11.694528 6250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:56:11.694562 6250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:56:11.695983 6250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:34305
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=43755
--tserver_master_addrs=127.23.167.254:42107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:56:11.696229 6250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:56:11.696437 6250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:56:11.701331 6260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:11.701341 6262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:11.701344 6259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:11.701579 6250 server_base.cc:1034] running on GCE node
I20260928 08:56:11.701785 6250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:56:11.702030 6250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:56:11.703202 6250 hybrid_clock.cc:648] HybridClock initialized: now 1790585771703186 us; error 24 us; skew 500 ppm
Sep 28 08:56:11 dist-test-slave-pbjp krb5kdc[5479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585771, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:11.705745 6250 init.cc:377] Logged in from keytab as kudu/127.23.167.194@KRBTEST.COM (short username kudu)
I20260928 08:56:11.706174 6250 webserver.cc:460] Webserver started at http://127.23.167.194:43755/ using document root <none> and password file <none>
I20260928 08:56:11.706388 6250 fs_manager.cc:359] Metadata directory not provided
I20260928 08:56:11.706453 6250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:56:11.707859 6250 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:56:11.708483 6269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:11.708668 6250 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:56:11.708752 6250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "00f838a4f4cc497bacd133ba8a2999b3"
format_stamp: "Formatted at 2026-09-28 08:56:10 on dist-test-slave-pbjp"
server_key: "4543a003133dbcb799b400a74c1f57d4"
server_key_iv: "883c88a66dcec23959f58efae8b44b2f"
server_key_version: "encryptionkey@0"
I20260928 08:56:11.709125 6250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:56:11.720716 6250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:56:11.722929 6250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:56:11.723074 6250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:56:11.723289 6250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:56:11.723735 6276 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 08:56:11.725180 6250 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 08:56:11.725231 6250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 08:56:11.725266 6250 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 08:56:11.725946 6250 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 08:56:11.725978 6250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 08:56:11.726029 6276 tablet_bootstrap.cc:492] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3: Bootstrap starting.
I20260928 08:56:11.732071 6250 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:34305
I20260928 08:56:11.732163 6383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:34305 every 8 connection(s)
I20260928 08:56:11.732481 6250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
I20260928 08:56:11.733891 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 6250
I20260928 08:56:11.734206 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:34325
--local_ip_for_outbound_sockets=127.23.167.195
--tserver_master_addrs=127.23.167.254:42107
--webserver_port=34753
--webserver_interface=127.23.167.195
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:56:11.735204 6276 log.cc:824] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3: Log is configured to *not* fsync() on all Append() calls
I20260928 08:56:11.737418 6276 tablet_bootstrap.cc:492] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:56:11.737711 6276 tablet_bootstrap.cc:492] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3: Bootstrap complete.
I20260928 08:56:11.738516 6276 ts_tablet_manager.cc:1397] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
Sep 28 08:56:11 dist-test-slave-pbjp krb5kdc[5479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585771, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:56:11.739717 6276 raft_consensus.cc:348] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 } } peers { permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 } } peers { permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 } }
I20260928 08:56:11.739986 6276 raft_consensus.cc:729] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00f838a4f4cc497bacd133ba8a2999b3, State: Initialized, Role: FOLLOWER
I20260928 08:56:11.740116 6276 consensus_queue.cc:260] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 } } peers { permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 } } peers { permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 } }
I20260928 08:56:11.740377 6276 ts_tablet_manager.cc:1428] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 08:56:11.742216 6248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:11.733248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:48181 (local address 127.23.167.254:42107)
0928 08:56:11.733318 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:11.733321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:11.733492 (+ 171us) server_negotiation.cc:407] Connection header received
0928 08:56:11.733981 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:11.733983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:11.733999 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:11.734045 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:11.734687 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.735195 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.736329 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.736541 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.736605 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:11.739150 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:11.739152 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:11.739152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:11.739161 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:11.740853 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:11.741277 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:11.741278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:11.741279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:11.741342 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:11.741581 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:11.741582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:11.741583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:11.741727 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:11.741799 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:11.742026 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 08:56:11.742125 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 08:56:11.742317 6386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:11.733183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.194:48181)
0928 08:56:11.733352 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:11.733396 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:11.733874 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:11.734127 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:11.734132 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:11.734399 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:56:11.734592 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:11.734596 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.735295 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.735296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:11.736245 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:11.736248 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.736631 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.736633 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:11.736762 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:11.737265 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:56:11.737275 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:56:11.739044 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:56:11.740931 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:11.740936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:11.740937 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:11.741211 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:11.741406 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:11.741407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:11.741407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:11.741502 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:11.741787 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:56:11.741791 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:56:11.741906 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 08:56:11.741988 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 08:56:11.742124 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
I20260928 08:56:11.742787 6384 heartbeater.cc:347] Connected to a master server at 127.23.167.254:42107
I20260928 08:56:11.742869 6384 heartbeater.cc:464] Registering TS with master...
I20260928 08:56:11.743027 6384 heartbeater.cc:511] Master 127.23.167.254:42107 requested a full tablet report, sending...
I20260928 08:56:11.743603 6057 ts_manager.cc:194] Registered new tserver with Master: 00f838a4f4cc497bacd133ba8a2999b3 (127.23.167.194:34305)
I20260928 08:56:11.744194 6057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.194@KRBTEST.COM'} at 127.23.167.194:48181
I20260928 08:56:11.744908 6384 heartbeater.cc:503] Master 127.23.167.254:42107 was elected leader, sending a full tablet report...
Sep 28 08:56:11 dist-test-slave-pbjp krb5kdc[5479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585771, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for kudu/127.23.167.194@KRBTEST.COM
I20260928 08:56:11.751190 6252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:11.743872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:34305 (local address 127.0.0.1:49678)
0928 08:56:11.743921 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:11.743930 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:11.743966 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:11.744311 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:11.744312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:11.744452 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:56:11.744498 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:11.744500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.745747 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.745748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:11.746245 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:11.746247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.746835 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.746836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:11.746878 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:11.747243 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:56:11.747248 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:56:11.748473 (+ 1225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:56:11.750241 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:11.750242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:11.750243 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:11.750386 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:11.750577 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:11.750578 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:11.750578 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:11.750631 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:11.750911 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:56:11.750911 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:56:11.750947 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 08:56:11.751035 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 08:56:11.751114 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 08:56:11.751269 6390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:11.743935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49678 (local address 127.23.167.194:34305)
0928 08:56:11.744080 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:11.744084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:11.744096 (+ 12us) server_negotiation.cc:407] Connection header received
0928 08:56:11.744139 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:11.744141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:11.744158 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:11.744248 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:11.744588 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.745673 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.746319 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.746772 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.746839 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:11.748551 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:11.748559 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:11.748559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:11.748567 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:11.750168 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:11.750453 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:11.750456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:11.750456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:11.750523 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:11.750680 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:11.750680 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:11.750681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:11.750852 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:11.750928 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:11.751069 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 08:56:11.751176 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":95,"threads_started":1}
I20260928 08:56:11.751595 6318 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fcdc12c61f34ed2a2c0084a8d89112f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:56:11 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:49678
I20260928 08:56:11.752084 6391 tablet_replica.cc:330] stopping tablet replica
I20260928 08:56:11.752192 6391 raft_consensus.cc:2229] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:56:11.752255 6391 raft_consensus.cc:2258] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:56:11.752537 6391 ts_tablet_manager.cc:1905] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:56:11.754464 6391 ts_tablet_manager.cc:1918] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:56:11.754530 6391 log.cc:1196] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/wal/wals/9fcdc12c61f34ed2a2c0084a8d89112f
I20260928 08:56:11.754716 6391 ts_tablet_manager.cc:1939] T 9fcdc12c61f34ed2a2c0084a8d89112f P 00f838a4f4cc497bacd133ba8a2999b3: Deleting consensus metadata
I20260928 08:56:11.755211 6042 catalog_manager.cc:4841] TS 00f838a4f4cc497bacd133ba8a2999b3 (127.23.167.194:34305): tablet 9fcdc12c61f34ed2a2c0084a8d89112f (table test-table [id=c90ffccf47c84e2ea88bb29cd05916a1]) successfully deleted
W20260928 08:56:11.820341 6388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:56:11.820555 6388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:56:11.820578 6388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:56:11.821799 6388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:56:11.821847 6388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:56:11.821875 6388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:56:11.823378 6388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:34325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=34753
--tserver_master_addrs=127.23.167.254:42107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:56:11.823601 6388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:56:11.823815 6388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:56:11.828564 6396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:11.828579 6397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:11.828653 6388 server_base.cc:1034] running on GCE node
W20260928 08:56:11.828543 6399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:11.829002 6388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:56:11.829259 6388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:56:11.830397 6388 hybrid_clock.cc:648] HybridClock initialized: now 1790585771830383 us; error 28 us; skew 500 ppm
Sep 28 08:56:11 dist-test-slave-pbjp krb5kdc[5479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585771, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:11.832753 6388 init.cc:377] Logged in from keytab as kudu/127.23.167.195@KRBTEST.COM (short username kudu)
I20260928 08:56:11.833231 6388 webserver.cc:460] Webserver started at http://127.23.167.195:34753/ using document root <none> and password file <none>
I20260928 08:56:11.833458 6388 fs_manager.cc:359] Metadata directory not provided
I20260928 08:56:11.833521 6388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:56:11.834797 6388 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:56:11.835348 6406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:11.835500 6388 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:56:11.835574 6388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "1f9e21caa9434b1f87809d96b2d26cfa"
format_stamp: "Formatted at 2026-09-28 08:56:10 on dist-test-slave-pbjp"
server_key: "e891a56e15f58bd3ca060536bc930ea7"
server_key_iv: "d17ace65586d2fb4a23592997540fc2a"
server_key_version: "encryptionkey@0"
I20260928 08:56:11.835906 6388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:56:11.864666 6388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:56:11.866930 6388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:56:11.867075 6388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:56:11.867316 6388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:56:11.867621 6388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:56:11.867655 6388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:11.867686 6388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:56:11.867707 6388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:11.873153 6388 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:34325
I20260928 08:56:11.873212 6519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:34325 every 8 connection(s)
I20260928 08:56:11.873543 6388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
Sep 28 08:56:11 dist-test-slave-pbjp krb5kdc[5479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585771, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:56:11.881664 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 6388
I20260928 08:56:11.882314 6248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:11.874212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:58765 (local address 127.23.167.254:42107)
0928 08:56:11.874273 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:11.874276 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:11.874491 (+ 215us) server_negotiation.cc:407] Connection header received
0928 08:56:11.875011 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:11.875012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:11.875024 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:11.875089 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:11.876035 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.876504 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.877445 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.877635 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.877682 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:11.879536 (+ 1854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:11.879537 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:11.879537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:11.879544 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:11.880801 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:11.881182 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:11.881183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:11.881183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:11.881231 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:11.881530 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:11.881531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:11.881532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:11.881703 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:11.881780 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:11.882144 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 08:56:11.882226 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 08:56:11.882491 6522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:11.874190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.195:58765)
0928 08:56:11.874359 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:11.874402 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:11.874891 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:11.875169 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:11.875172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:11.875738 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:56:11.875951 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:11.875957 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.876562 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.876563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:11.877369 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:11.877371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.877690 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.877690 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:11.877783 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:11.878191 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:56:11.878199 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:56:11.879456 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:56:11.880871 (+ 1415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:11.880877 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:11.880879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:11.881124 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:11.881278 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:11.881278 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:11.881279 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:11.881374 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:11.881792 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:56:11.881799 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:56:11.881948 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 08:56:11.882120 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 08:56:11.882279 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
I20260928 08:56:11.882929 6520 heartbeater.cc:347] Connected to a master server at 127.23.167.254:42107
I20260928 08:56:11.883018 6520 heartbeater.cc:464] Registering TS with master...
I20260928 08:56:11.883189 6520 heartbeater.cc:511] Master 127.23.167.254:42107 requested a full tablet report, sending...
I20260928 08:56:11.883541 6057 ts_manager.cc:194] Registered new tserver with Master: 1f9e21caa9434b1f87809d96b2d26cfa (127.23.167.195:34325)
I20260928 08:56:11.883886 6057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.195@KRBTEST.COM'} at 127.23.167.195:58765
I20260928 08:56:11.887816 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:11.881993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47958 (local address 127.23.167.254:42107)
0928 08:56:11.882104 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:11.882106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:11.882140 (+ 34us) server_negotiation.cc:407] Connection header received
0928 08:56:11.882216 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:11.882216 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:11.882227 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:11.882288 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:11.882932 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.883524 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.884289 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.884492 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.884544 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:11.885635 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:11.885636 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:11.885636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:11.885643 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:11.886866 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:11.887155 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:11.887156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:11.887156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:11.887209 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:11.887363 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:11.887364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:11.887365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:11.887465 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:11.887610 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:11.887650 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 08:56:11.887725 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":60,"threads_started":1}
I20260928 08:56:11.888432 24223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:56:11.890786 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:11.888761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47974 (local address 127.23.167.254:42107)
0928 08:56:11.888801 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:11.888803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:11.888859 (+ 56us) server_negotiation.cc:407] Connection header received
0928 08:56:11.888909 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:11.888910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:11.888935 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:11.888987 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:56:11.889053 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.889556 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.890227 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.890412 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.890459 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:11.890552 (+ 93us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:56:11.890652 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:56:11.890707 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:11.890720 (+ 13us) server_negotiation.cc:299] Negotiation successful
0928 08:56:11.890736 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 08:56:11.891034 6057 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:47974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:56:11.892588 6057 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:56:11.894570 6179 tablet_service.cc:1467] Processing CreateTablet for tablet 1b1219c3c1e642229dcc0a0a9374b817 (DEFAULT_TABLE table=test-table [id=08bf72c6bf8e4dd48352fe03d099718d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:56:11.894562 6318 tablet_service.cc:1467] Processing CreateTablet for tablet 1b1219c3c1e642229dcc0a0a9374b817 (DEFAULT_TABLE table=test-table [id=08bf72c6bf8e4dd48352fe03d099718d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:56:11.894742 6318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b1219c3c1e642229dcc0a0a9374b817. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:56:11.894742 6179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b1219c3c1e642229dcc0a0a9374b817. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:56:11.896209 6276 tablet_bootstrap.cc:492] T 1b1219c3c1e642229dcc0a0a9374b817 P 00f838a4f4cc497bacd133ba8a2999b3: Bootstrap starting.
I20260928 08:56:11.896533 6276 tablet_bootstrap.cc:654] T 1b1219c3c1e642229dcc0a0a9374b817 P 00f838a4f4cc497bacd133ba8a2999b3: Neither blocks nor log segments found. Creating new log.
I20260928 08:56:11.896793 6137 tablet_bootstrap.cc:492] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f: Bootstrap starting.
I20260928 08:56:11.897096 6137 tablet_bootstrap.cc:654] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f: Neither blocks nor log segments found. Creating new log.
I20260928 08:56:11.897107 6276 tablet_bootstrap.cc:492] T 1b1219c3c1e642229dcc0a0a9374b817 P 00f838a4f4cc497bacd133ba8a2999b3: No bootstrap required, opened a new log
I20260928 08:56:11.897145 6276 ts_tablet_manager.cc:1397] T 1b1219c3c1e642229dcc0a0a9374b817 P 00f838a4f4cc497bacd133ba8a2999b3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 08:56:11.897481 6276 raft_consensus.cc:348] T 1b1219c3c1e642229dcc0a0a9374b817 P 00f838a4f4cc497bacd133ba8a2999b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 } } peers { permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 } } peers { permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 } }
I20260928 08:56:11.897553 6276 raft_consensus.cc:374] T 1b1219c3c1e642229dcc0a0a9374b817 P 00f838a4f4cc497bacd133ba8a2999b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:56:11.897598 6276 raft_consensus.cc:729] T 1b1219c3c1e642229dcc0a0a9374b817 P 00f838a4f4cc497bacd133ba8a2999b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00f838a4f4cc497bacd133ba8a2999b3, State: Initialized, Role: FOLLOWER
I20260928 08:56:11.897610 6137 tablet_bootstrap.cc:492] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f: No bootstrap required, opened a new log
I20260928 08:56:11.897644 6137 ts_tablet_manager.cc:1397] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 08:56:11.897644 6276 consensus_queue.cc:260] T 1b1219c3c1e642229dcc0a0a9374b817 P 00f838a4f4cc497bacd133ba8a2999b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 } } peers { permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 } } peers { permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 } }
I20260928 08:56:11.897739 6276 ts_tablet_manager.cc:1428] T 1b1219c3c1e642229dcc0a0a9374b817 P 00f838a4f4cc497bacd133ba8a2999b3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 08:56:11.897861 6252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:11.894328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:34325 (local address 127.0.0.1:45810)
0928 08:56:11.894368 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:11.894377 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:11.894408 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:11.894814 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:11.894815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:11.894817 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:56:11.894856 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:11.894857 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.895748 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.895749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:11.896679 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:11.896683 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.897624 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.897625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:11.897691 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:11.897711 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 08:56:11.897760 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 08:56:11.897800 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 08:56:11.897967 6137 raft_consensus.cc:348] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 } } peers { permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 } } peers { permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 } }
I20260928 08:56:11.898034 6137 raft_consensus.cc:374] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:56:11.898069 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:11.894485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45810 (local address 127.23.167.195:34325)
0928 08:56:11.894666 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:11.894671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:11.894681 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:56:11.894712 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:11.894714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:11.894726 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:11.894797 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:56:11.894915 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.895663 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.896769 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:11.897549 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:11.897618 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:11.897677 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:11.897953 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 08:56:11.897996 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":96,"threads_started":1}
I20260928 08:56:11.898092 6137 raft_consensus.cc:729] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d135091233e4b609b1a7a67b7e5278f, State: Initialized, Role: FOLLOWER
I20260928 08:56:11.898135 6137 consensus_queue.cc:260] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 } } peers { permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 } } peers { permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 } }
I20260928 08:56:11.898212 6137 ts_tablet_manager.cc:1428] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 08:56:11.899044 6454 tablet_service.cc:1467] Processing CreateTablet for tablet 1b1219c3c1e642229dcc0a0a9374b817 (DEFAULT_TABLE table=test-table [id=08bf72c6bf8e4dd48352fe03d099718d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:56:11.899322 6454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b1219c3c1e642229dcc0a0a9374b817. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:56:11.901928 6529 tablet_bootstrap.cc:492] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa: Bootstrap starting.
I20260928 08:56:11.902542 6529 tablet_bootstrap.cc:654] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa: Neither blocks nor log segments found. Creating new log.
I20260928 08:56:11.902974 6529 log.cc:824] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa: Log is configured to *not* fsync() on all Append() calls
I20260928 08:56:11.903625 6529 tablet_bootstrap.cc:492] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa: No bootstrap required, opened a new log
I20260928 08:56:11.903679 6529 ts_tablet_manager.cc:1397] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:56:11.904985 6529 raft_consensus.cc:348] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 } } peers { permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 } } peers { permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 } }
I20260928 08:56:11.905165 6529 raft_consensus.cc:374] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:56:11.905200 6529 raft_consensus.cc:729] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9e21caa9434b1f87809d96b2d26cfa, State: Initialized, Role: FOLLOWER
I20260928 08:56:11.905313 6529 consensus_queue.cc:260] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 } } peers { permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 } } peers { permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 } }
I20260928 08:56:11.905568 6529 ts_tablet_manager.cc:1428] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:56:11.905576 6520 heartbeater.cc:503] Master 127.23.167.254:42107 was elected leader, sending a full tablet report...
W20260928 08:56:11.983103 6385 tablet.cc:2369] T 1b1219c3c1e642229dcc0a0a9374b817 P 00f838a4f4cc497bacd133ba8a2999b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:56:12.057001 6531 raft_consensus.cc:482] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:56:12.057168 6531 raft_consensus.cc:504] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 } } peers { permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 } } peers { permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 } }
I20260928 08:56:12.057536 6531 leader_election.cc:288] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00f838a4f4cc497bacd133ba8a2999b3 (127.23.167.194:34305), 6d135091233e4b609b1a7a67b7e5278f (127.23.167.193:34943)
I20260928 08:56:12.060369 6390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:12.057826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:38881 (local address 127.23.167.194:34305)
0928 08:56:12.057891 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:12.057892 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:12.057932 (+ 40us) server_negotiation.cc:407] Connection header received
0928 08:56:12.057958 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:12.057959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:12.057981 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:12.058039 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:56:12.058204 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:12.058707 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:12.059611 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:12.060046 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:12.060121 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:12.060205 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:12.060262 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 08:56:12.060300 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 08:56:12.060369 6522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:12.057770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:34305 (local address 127.23.167.195:38881)
0928 08:56:12.057816 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:12.057828 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:12.057873 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:12.058057 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:12.058058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:12.058060 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:56:12.058132 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:12.058134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:12.058784 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:12.058785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:12.059513 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:12.059515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:12.060130 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:12.060131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:12.060206 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:12.060210 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:56:12.060258 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 08:56:12.060298 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 08:56:12.060593 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:12.057770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:34943 (local address 127.23.167.195:47087)
0928 08:56:12.058069 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:12.058076 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:12.058122 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:12.058272 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:12.058272 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:12.058274 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:56:12.058338 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:12.058340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:12.059043 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:12.059044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:12.059776 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:12.059778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:12.060338 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:12.060339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:12.060406 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:12.060410 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:56:12.060467 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 08:56:12.060509 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":166,"thread_start_us":75,"threads_started":1}
I20260928 08:56:12.060607 6253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:12.057827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:47087 (local address 127.23.167.193:34943)
0928 08:56:12.057876 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:12.057878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:12.058141 (+ 263us) server_negotiation.cc:407] Connection header received
0928 08:56:12.058173 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:12.058174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:12.058201 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:12.058255 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:56:12.058448 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:12.058961 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:12.059862 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:12.060257 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:12.060318 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:12.060384 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:12.060505 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 08:56:12.060547 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 08:56:12.060856 6338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b1219c3c1e642229dcc0a0a9374b817" candidate_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00f838a4f4cc497bacd133ba8a2999b3" is_pre_election: true
I20260928 08:56:12.060861 6199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b1219c3c1e642229dcc0a0a9374b817" candidate_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d135091233e4b609b1a7a67b7e5278f" is_pre_election: true
I20260928 08:56:12.060997 6338 raft_consensus.cc:2454] T 1b1219c3c1e642229dcc0a0a9374b817 P 00f838a4f4cc497bacd133ba8a2999b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9e21caa9434b1f87809d96b2d26cfa in term 0.
I20260928 08:56:12.061002 6199 raft_consensus.cc:2454] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9e21caa9434b1f87809d96b2d26cfa in term 0.
I20260928 08:56:12.061235 6410 leader_election.cc:302] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9e21caa9434b1f87809d96b2d26cfa, 6d135091233e4b609b1a7a67b7e5278f; no voters:
I20260928 08:56:12.061350 6531 raft_consensus.cc:2781] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:56:12.061411 6531 raft_consensus.cc:482] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:56:12.061427 6531 raft_consensus.cc:3037] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:56:12.062083 6531 raft_consensus.cc:504] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 } } peers { permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 } } peers { permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 } }
I20260928 08:56:12.062191 6531 leader_election.cc:288] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa [CANDIDATE]: Term 1 election: Requested vote from peers 00f838a4f4cc497bacd133ba8a2999b3 (127.23.167.194:34305), 6d135091233e4b609b1a7a67b7e5278f (127.23.167.193:34943)
I20260928 08:56:12.062325 6199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b1219c3c1e642229dcc0a0a9374b817" candidate_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d135091233e4b609b1a7a67b7e5278f"
I20260928 08:56:12.062345 6338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b1219c3c1e642229dcc0a0a9374b817" candidate_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00f838a4f4cc497bacd133ba8a2999b3"
I20260928 08:56:12.062402 6199 raft_consensus.cc:3037] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:56:12.062410 6338 raft_consensus.cc:3037] T 1b1219c3c1e642229dcc0a0a9374b817 P 00f838a4f4cc497bacd133ba8a2999b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:56:12.063076 6199 raft_consensus.cc:2454] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9e21caa9434b1f87809d96b2d26cfa in term 1.
I20260928 08:56:12.063095 6338 raft_consensus.cc:2454] T 1b1219c3c1e642229dcc0a0a9374b817 P 00f838a4f4cc497bacd133ba8a2999b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9e21caa9434b1f87809d96b2d26cfa in term 1.
I20260928 08:56:12.063247 6410 leader_election.cc:302] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9e21caa9434b1f87809d96b2d26cfa, 6d135091233e4b609b1a7a67b7e5278f; no voters:
I20260928 08:56:12.063356 6531 raft_consensus.cc:2781] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:56:12.063493 6531 raft_consensus.cc:686] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa [term 1 LEADER]: Becoming Leader. State: Replica: 1f9e21caa9434b1f87809d96b2d26cfa, State: Running, Role: LEADER
I20260928 08:56:12.063580 6531 consensus_queue.cc:237] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 } } peers { permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 } } peers { permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 } }
I20260928 08:56:12.064364 6056 catalog_manager.cc:5495] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f9e21caa9434b1f87809d96b2d26cfa (127.23.167.195). New cstate: current_term: 1 leader_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 } health_report { overall_health: UNKNOWN } } }
I20260928 08:56:12.081781 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:12.079581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45816 (local address 127.23.167.195:34325)
0928 08:56:12.079646 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:12.079647 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:12.079656 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:56:12.079719 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:12.079720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:12.079737 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:12.079793 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:56:12.079884 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:12.080348 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:12.081023 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:12.081275 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:12.081341 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:12.081456 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:56:12.081574 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:56:12.081654 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:12.081669 (+ 15us) server_negotiation.cc:299] Negotiation successful
0928 08:56:12.081685 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 08:56:12.083482 6338 raft_consensus.cc:1261] T 1b1219c3c1e642229dcc0a0a9374b817 P 00f838a4f4cc497bacd133ba8a2999b3 [term 1 FOLLOWER]: Refusing update from remote peer 1f9e21caa9434b1f87809d96b2d26cfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:56:12.083524 6199 raft_consensus.cc:1261] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [term 1 FOLLOWER]: Refusing update from remote peer 1f9e21caa9434b1f87809d96b2d26cfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:56:12.083879 6533 consensus_queue.cc:1035] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:56:12.084000 6531 consensus_queue.cc:1035] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:56:12.086004 6538 mvcc.cc:205] Tried to move back new op lower bound from 7334239322451505152 to 7334239322373013504. Current Snapshot: MvccSnapshot[applied={T|T < 7334239322451505152}]
I20260928 08:56:12.086297 6537 mvcc.cc:205] Tried to move back new op lower bound from 7334239322451505152 to 7334239322373013504. Current Snapshot: MvccSnapshot[applied={T|T < 7334239322451505152}]
I20260928 08:56:12.088140 6057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:47974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:56:12.088217 6057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:47974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:56:12.088840 6057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0: Sending DeleteTablet for 3 replicas of tablet 1b1219c3c1e642229dcc0a0a9374b817
I20260928 08:56:12.089157 6179 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b1219c3c1e642229dcc0a0a9374b817 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:56:12 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:59490
I20260928 08:56:12.089141 6454 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b1219c3c1e642229dcc0a0a9374b817 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:56:12 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:45810
I20260928 08:56:12.089265 6254 tablet_replica.cc:330] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f: stopping tablet replica
I20260928 08:56:12.089334 6254 raft_consensus.cc:2229] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:56:12.089351 6318 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b1219c3c1e642229dcc0a0a9374b817 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:56:12 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:49678
I20260928 08:56:12.089394 6254 raft_consensus.cc:2258] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:56:12.089437 6391 tablet_replica.cc:330] T 1b1219c3c1e642229dcc0a0a9374b817 P 00f838a4f4cc497bacd133ba8a2999b3: stopping tablet replica
I20260928 08:56:12.089478 6391 raft_consensus.cc:2229] T 1b1219c3c1e642229dcc0a0a9374b817 P 00f838a4f4cc497bacd133ba8a2999b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:56:12.089511 6391 raft_consensus.cc:2258] T 1b1219c3c1e642229dcc0a0a9374b817 P 00f838a4f4cc497bacd133ba8a2999b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:56:12.089586 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 6095
I20260928 08:56:12.089783 6254 ts_tablet_manager.cc:1905] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:56:12.090459 6545 tablet_replica.cc:330] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa: stopping tablet replica
I20260928 08:56:12.090550 6545 raft_consensus.cc:2229] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:56:12.090670 6545 raft_consensus.cc:2258] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:56:12.093876 6391 ts_tablet_manager.cc:1905] T 1b1219c3c1e642229dcc0a0a9374b817 P 00f838a4f4cc497bacd133ba8a2999b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 08:56:12.093931 6044 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.167.193:34943 (error 108)
I20260928 08:56:12.094455 6545 ts_tablet_manager.cc:1905] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:56:12.094890 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 6250
I20260928 08:56:12.096374 6545 ts_tablet_manager.cc:1918] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:56:12.096450 6545 log.cc:1196] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/wal/wals/1b1219c3c1e642229dcc0a0a9374b817
I20260928 08:56:12.096897 6252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:12.096767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:34943 (local address 127.0.0.1:59506)
0928 08:56:12.096821 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:12.096863 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.193:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 08:56:12.097007 6044 catalog_manager.cc:4584] TS 6d135091233e4b609b1a7a67b7e5278f (127.23.167.193:34943): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1b1219c3c1e642229dcc0a0a9374b817: Network error: Client connection negotiation failed: client connection to 127.23.167.193:34943: connect: Connection refused (error 111)
I20260928 08:56:12.097533 6545 ts_tablet_manager.cc:1939] T 1b1219c3c1e642229dcc0a0a9374b817 P 1f9e21caa9434b1f87809d96b2d26cfa: Deleting consensus metadata
I20260928 08:56:12.097915 6041 catalog_manager.cc:4841] TS 1f9e21caa9434b1f87809d96b2d26cfa (127.23.167.195:34325): tablet 1b1219c3c1e642229dcc0a0a9374b817 (table test-table [id=08bf72c6bf8e4dd48352fe03d099718d]) successfully deleted
I20260928 08:56:12.100625 6252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:12.100522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:34305 (local address 127.0.0.1:49680)
0928 08:56:12.100565 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:12.100589 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.194:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 08:56:12.100754 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 6388
I20260928 08:56:12.105513 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 6024
I20260928 08:56:12.110020 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:42107
--webserver_interface=127.23.167.254
--webserver_port=44571
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:42107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:56:12.194448 6547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:56:12.194684 6547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:56:12.194707 6547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:56:12.195816 6547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:56:12.195854 6547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:56:12.195866 6547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:56:12.195878 6547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:56:12.195888 6547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:56:12.197332 6547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:42107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:42107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=44571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:56:12.197530 6547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:56:12.197712 6547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:56:12.202548 6553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:12.202646 6552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:12.202600 6547 server_base.cc:1034] running on GCE node
W20260928 08:56:12.202543 6555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:12.203029 6547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:56:12.203446 6547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:56:12.204587 6547 hybrid_clock.cc:648] HybridClock initialized: now 1790585772204563 us; error 41 us; skew 500 ppm
Sep 28 08:56:12 dist-test-slave-pbjp krb5kdc[5479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585772, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:12.207124 6547 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:56:12.207530 6547 webserver.cc:460] Webserver started at http://127.23.167.254:44571/ using document root <none> and password file <none>
I20260928 08:56:12.207743 6547 fs_manager.cc:359] Metadata directory not provided
I20260928 08:56:12.207792 6547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:56:12.209044 6547 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:56:12.209681 6562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:12.209820 6547 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:56:12.209898 6547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "7c14df0884684549b1c32bec060cd0e0"
format_stamp: "Formatted at 2026-09-28 08:56:10 on dist-test-slave-pbjp"
server_key: "31eef92ae8608ff691b1846a873e5713"
server_key_iv: "b1e7423693030f1a3b589ff2e666d235"
server_key_version: "encryptionkey@0"
I20260928 08:56:12.210240 6547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:56:12.226032 6547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:56:12.227948 6547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:56:12.228082 6547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:56:12.231976 6547 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:42107
I20260928 08:56:12.232024 6614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:42107 every 8 connection(s)
I20260928 08:56:12.232451 6547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:56:12.234066 6615 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:56:12.234660 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 6547
I20260928 08:56:12.234850 6615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0: Bootstrap starting.
I20260928 08:56:12.234951 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:34943
--local_ip_for_outbound_sockets=127.23.167.193
--tserver_master_addrs=127.23.167.254:42107
--webserver_port=41551
--webserver_interface=127.23.167.193
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:56:12.244225 6615 log.cc:824] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0: Log is configured to *not* fsync() on all Append() calls
I20260928 08:56:12.246618 6615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:56:12.247217 6615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0: Bootstrap complete.
I20260928 08:56:12.249449 6615 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c14df0884684549b1c32bec060cd0e0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42107 } }
I20260928 08:56:12.249688 6615 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c14df0884684549b1c32bec060cd0e0, State: Initialized, Role: FOLLOWER
I20260928 08:56:12.249835 6615 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c14df0884684549b1c32bec060cd0e0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42107 } }
I20260928 08:56:12.249897 6615 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:56:12.249938 6615 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:56:12.249982 6615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [term 2 FOLLOWER]: Advancing to term 3
I20260928 08:56:12.250946 6615 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c14df0884684549b1c32bec060cd0e0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42107 } }
I20260928 08:56:12.251039 6615 leader_election.cc:302] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c14df0884684549b1c32bec060cd0e0; no voters:
I20260928 08:56:12.251200 6615 leader_election.cc:288] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 08:56:12.251256 6619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 08:56:12.251368 6619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [term 3 LEADER]: Becoming Leader. State: Replica: 7c14df0884684549b1c32bec060cd0e0, State: Running, Role: LEADER
I20260928 08:56:12.251483 6615 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:56:12.251452 6619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c14df0884684549b1c32bec060cd0e0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42107 } }
I20260928 08:56:12.251760 6619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c14df0884684549b1c32bec060cd0e0. Latest consensus state: current_term: 3 leader_uuid: "7c14df0884684549b1c32bec060cd0e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c14df0884684549b1c32bec060cd0e0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42107 } } }
I20260928 08:56:12.251824 6619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [sys.catalog]: This master's current role is: LEADER
I20260928 08:56:12.251981 6621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7c14df0884684549b1c32bec060cd0e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c14df0884684549b1c32bec060cd0e0" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42107 } } }
I20260928 08:56:12.252024 6623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:56:12.252049 6621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c14df0884684549b1c32bec060cd0e0 [sys.catalog]: This master's current role is: LEADER
I20260928 08:56:12.253000 6623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:56:12.253264 6623 catalog_manager.cc:1259] Loaded cluster ID: f0867759b8b3417fb40da338becb8163
I20260928 08:56:12.253549 6623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 08:56:12.253860 6623 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f5947f889ad google::LogMessage::Fail() at ??:0
@ 0x7f5947f8b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f5947f884de google::LogMessage::Flush() at ??:0
@ 0x7f5947f89a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f594fea90f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f594feb28b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f5949e560d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5949e4b954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f594acaf6db start_thread at ??:0
@ 0x7f5946a8661f clone at ??:0
W20260928 08:56:12.329152 6617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:56:12.329347 6617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:56:12.329375 6617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:56:12.330514 6617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:56:12.330555 6617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:56:12.330581 6617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:56:12.332043 6617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:34943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=41551
--tserver_master_addrs=127.23.167.254:42107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:56:12.332265 6617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:56:12.332473 6617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:56:12.337255 6652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:12.337260 6651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:12.337339 6617 server_base.cc:1034] running on GCE node
W20260928 08:56:12.337427 6654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:12.337728 6617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:56:12.337968 6617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:56:12.339124 6617 hybrid_clock.cc:648] HybridClock initialized: now 1790585772339112 us; error 26 us; skew 500 ppm
Sep 28 08:56:12 dist-test-slave-pbjp krb5kdc[5479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585772, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:12.341540 6617 init.cc:377] Logged in from keytab as kudu/127.23.167.193@KRBTEST.COM (short username kudu)
I20260928 08:56:12.341984 6617 webserver.cc:460] Webserver started at http://127.23.167.193:41551/ using document root <none> and password file <none>
I20260928 08:56:12.342167 6617 fs_manager.cc:359] Metadata directory not provided
I20260928 08:56:12.342206 6617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:56:12.343391 6617 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:56:12.344067 6661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:12.344234 6617 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:12.344336 6617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "6d135091233e4b609b1a7a67b7e5278f"
format_stamp: "Formatted at 2026-09-28 08:56:10 on dist-test-slave-pbjp"
server_key: "b8a4c345b7642c56e6507e056631274b"
server_key_iv: "07643772ecf3f758cbbe569f3a01d446"
server_key_version: "encryptionkey@0"
I20260928 08:56:12.344820 6617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:56:12.360003 6617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:56:12.362136 6617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:56:12.362286 6617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:56:12.362540 6617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:56:12.362942 6668 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 08:56:12.364312 6617 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 08:56:12.364372 6617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 08:56:12.364406 6617 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 08:56:12.365326 6617 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 08:56:12.365372 6617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260928 08:56:12.365430 6668 tablet_bootstrap.cc:492] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f: Bootstrap starting.
I20260928 08:56:12.366751 6668 log.cc:824] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f: Log is configured to *not* fsync() on all Append() calls
I20260928 08:56:12.372936 6668 tablet_bootstrap.cc:492] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:56:12.373886 6668 tablet_bootstrap.cc:492] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f: Bootstrap complete.
I20260928 08:56:12.374320 6668 ts_tablet_manager.cc:1397] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20260928 08:56:12.377521 6668 raft_consensus.cc:348] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 } } peers { permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 } } peers { permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 } }
I20260928 08:56:12.377959 6668 raft_consensus.cc:729] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d135091233e4b609b1a7a67b7e5278f, State: Initialized, Role: FOLLOWER
I20260928 08:56:12.378314 6668 consensus_queue.cc:260] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 } } peers { permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 } } peers { permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 } }
I20260928 08:56:12.378880 6668 ts_tablet_manager.cc:1428] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f: Time spent starting tablet: real 0.004s user 0.003s sys 0.003s
I20260928 08:56:12.382381 6617 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:34943
I20260928 08:56:12.382442 6777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:34943 every 8 connection(s)
I20260928 08:56:12.383311 6617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
I20260928 08:56:12.390703 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 6617
I20260928 08:56:12.391054 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:34305
--local_ip_for_outbound_sockets=127.23.167.194
--tserver_master_addrs=127.23.167.254:42107
--webserver_port=43755
--webserver_interface=127.23.167.194
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:56:12.536465 6781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:56:12.536826 6781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:56:12.536876 6781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:56:12.538780 6781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:56:12.538849 6781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:56:12.538918 6781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:56:12.541337 6781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:34305
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=43755
--tserver_master_addrs=127.23.167.254:42107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:56:12.541785 6781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:56:12.542289 6781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:56:12.547674 6780 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:56:12.385154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.193:39673)
0928 08:56:12.385520 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:12.385572 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:12.385874 (+ 302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:12.547287 (+161413us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":232,"threads_started":1}
W20260928 08:56:12.548019 6778 heartbeater.cc:644] Failed to heartbeat to 127.23.167.254:42107 (0 consecutive failures): Network error: Failed to ping master at 127.23.167.254:42107: Client connection negotiation failed: client connection to 127.23.167.254:42107: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 08:56:12.548528 6780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:12.548358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.193:45935)
0928 08:56:12.548431 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:12.548493 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 08:56:12.549603 6780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:12.549444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.193:53723)
0928 08:56:12.549520 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:12.549548 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 08:56:12.549902 6778 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 08:56:12.550393 6789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:12.550395 6787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:12.550395 6786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:12.551141 6781 server_base.cc:1034] running on GCE node
I20260928 08:56:12.551470 6781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:56:12.551852 6781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:56:12.553041 6781 hybrid_clock.cc:648] HybridClock initialized: now 1790585772553013 us; error 49 us; skew 500 ppm
Sep 28 08:56:12 dist-test-slave-pbjp krb5kdc[5479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585772, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:12.558688 6781 init.cc:377] Logged in from keytab as kudu/127.23.167.194@KRBTEST.COM (short username kudu)
I20260928 08:56:12.559471 6781 webserver.cc:460] Webserver started at http://127.23.167.194:43755/ using document root <none> and password file <none>
I20260928 08:56:12.559798 6781 fs_manager.cc:359] Metadata directory not provided
I20260928 08:56:12.559875 6781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:56:12.561986 6781 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 08:56:12.562948 6796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:12.563182 6781 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:56:12.563305 6781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "00f838a4f4cc497bacd133ba8a2999b3"
format_stamp: "Formatted at 2026-09-28 08:56:10 on dist-test-slave-pbjp"
server_key: "4543a003133dbcb799b400a74c1f57d4"
server_key_iv: "883c88a66dcec23959f58efae8b44b2f"
server_key_version: "encryptionkey@0"
I20260928 08:56:12.563848 6781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:56:12.586220 6781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:56:12.589627 6781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:56:12.589910 6781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:56:12.590314 6781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:56:12.590991 6803 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260928 08:56:12.593530 6803 ts_tablet_manager.cc:1852] T 1b1219c3c1e642229dcc0a0a9374b817 P 00f838a4f4cc497bacd133ba8a2999b3: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 08:56:12.594008 6803 ts_tablet_manager.cc:1905] T 1b1219c3c1e642229dcc0a0a9374b817 P 00f838a4f4cc497bacd133ba8a2999b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:56:12.597038 6803 ts_tablet_manager.cc:1918] T 1b1219c3c1e642229dcc0a0a9374b817 P 00f838a4f4cc497bacd133ba8a2999b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:56:12.597208 6803 log.cc:1196] T 1b1219c3c1e642229dcc0a0a9374b817 P 00f838a4f4cc497bacd133ba8a2999b3: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/wal/wals/1b1219c3c1e642229dcc0a0a9374b817
I20260928 08:56:12.597671 6803 ts_tablet_manager.cc:1939] T 1b1219c3c1e642229dcc0a0a9374b817 P 00f838a4f4cc497bacd133ba8a2999b3: Deleting consensus metadata
I20260928 08:56:12.598011 6781 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 08:56:12.598084 6781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.000s
I20260928 08:56:12.598171 6781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:56:12.598233 6781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:12.610744 6781 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:34305
I20260928 08:56:12.610848 6910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:34305 every 8 connection(s)
I20260928 08:56:12.611363 6781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
I20260928 08:56:12.612823 6913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:12.612299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.194:45357)
0928 08:56:12.612505 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:12.612618 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":94,"threads_started":1}
W20260928 08:56:12.613130 6911 heartbeater.cc:644] Failed to heartbeat to 127.23.167.254:42107 (0 consecutive failures): Network error: Failed to ping master at 127.23.167.254:42107: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
I20260928 08:56:12.613498 6913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:12.613380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.194:57771)
0928 08:56:12.613441 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:12.613475 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 08:56:12.613837 6913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:12.613749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.194:50059)
0928 08:56:12.613796 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:12.613824 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 08:56:12.613940 6911 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 08:56:12.619874 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 6781
I20260928 08:56:12.620137 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:34325
--local_ip_for_outbound_sockets=127.23.167.195
--tserver_master_addrs=127.23.167.254:42107
--webserver_port=34753
--webserver_interface=127.23.167.195
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:56:12.781968 6914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:56:12.782169 6914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:56:12.782191 6914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:56:12.783375 6914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:56:12.783413 6914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:56:12.783439 6914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:56:12.784870 6914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:34325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=34753
--tserver_master_addrs=127.23.167.254:42107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:56:12.785112 6914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:56:12.785331 6914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 08:56:12.790009 6914 server_base.cc:1034] running on GCE node
W20260928 08:56:12.790014 6919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:12.789983 6920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:12.789983 6922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:12.790378 6914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:56:12.790611 6914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:56:12.791749 6914 hybrid_clock.cc:648] HybridClock initialized: now 1790585772791740 us; error 25 us; skew 500 ppm
Sep 28 08:56:12 dist-test-slave-pbjp krb5kdc[5479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585772, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:12.794013 6914 init.cc:377] Logged in from keytab as kudu/127.23.167.195@KRBTEST.COM (short username kudu)
I20260928 08:56:12.794399 6914 webserver.cc:460] Webserver started at http://127.23.167.195:34753/ using document root <none> and password file <none>
I20260928 08:56:12.794606 6914 fs_manager.cc:359] Metadata directory not provided
I20260928 08:56:12.794653 6914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:56:12.795863 6914 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:56:12.796339 6929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:12.796479 6914 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:12.796594 6914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "1f9e21caa9434b1f87809d96b2d26cfa"
format_stamp: "Formatted at 2026-09-28 08:56:10 on dist-test-slave-pbjp"
server_key: "e891a56e15f58bd3ca060536bc930ea7"
server_key_iv: "d17ace65586d2fb4a23592997540fc2a"
server_key_version: "encryptionkey@0"
I20260928 08:56:12.796896 6914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:56:12.813696 6914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:56:12.815439 6914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:56:12.815570 6914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:56:12.815788 6914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:56:12.816068 6914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:56:12.816104 6914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:12.816134 6914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:56:12.816157 6914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:12.821702 6914 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:34325
I20260928 08:56:12.821753 7042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:34325 every 8 connection(s)
I20260928 08:56:12.822072 6914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
I20260928 08:56:12.823041 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:12.822676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.195:45937)
0928 08:56:12.822820 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:12.822898 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
W20260928 08:56:12.823393 7043 heartbeater.cc:644] Failed to heartbeat to 127.23.167.254:42107 (0 consecutive failures): Network error: Failed to ping master at 127.23.167.254:42107: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
I20260928 08:56:12.823632 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:12.823550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.195:59891)
0928 08:56:12.823585 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:12.823616 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 08:56:12.823837 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:12.823776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.195:38015)
0928 08:56:12.823805 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:12.823822 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260928 08:56:12.823932 7043 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 08:56:12.829677 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 6914
I20260928 08:56:13.550431 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:13.550214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.193:48107)
0928 08:56:13.550331 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:13.550369 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260928 08:56:13.614459 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:13.614273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.194:51543)
0928 08:56:13.614384 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:13.614422 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":55,"threads_started":1}
I20260928 08:56:13.824443 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:13.824243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.195:48057)
0928 08:56:13.824357 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:13.824397 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":59,"threads_started":1}
I20260928 08:56:13.894658 7050 raft_consensus.cc:482] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:56:13.894796 7050 raft_consensus.cc:504] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 } } peers { permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 } } peers { permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 } }
I20260928 08:56:13.895119 7050 leader_election.cc:288] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f9e21caa9434b1f87809d96b2d26cfa (127.23.167.195:34325), 00f838a4f4cc497bacd133ba8a2999b3 (127.23.167.194:34305)
Sep 28 08:56:13 dist-test-slave-pbjp krb5kdc[5479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585772, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for kudu/127.23.167.194@KRBTEST.COM
Sep 28 08:56:13 dist-test-slave-pbjp krb5kdc[5479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585772, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for kudu/127.23.167.195@KRBTEST.COM
I20260928 08:56:13.905828 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:13.895299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:34305 (local address 127.23.167.193:50457)
0928 08:56:13.895360 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:13.895370 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:13.895421 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:13.896482 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:13.896482 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:13.896941 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:56:13.897400 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:13.897407 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:13.898485 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:13.898486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:13.899428 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:13.899431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:13.900124 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:13.900125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:13.900254 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:13.900771 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:56:13.900779 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:56:13.902459 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:56:13.904264 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:13.904272 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:13.904274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:13.904562 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:13.904799 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:13.904799 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:13.904800 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:13.904933 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:13.905309 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:56:13.905316 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:56:13.905484 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 08:56:13.905569 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 08:56:13.905714 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 08:56:13.905869 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:13.895379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:50457 (local address 127.23.167.194:34305)
0928 08:56:13.895543 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:13.895557 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:13.895582 (+ 25us) server_negotiation.cc:407] Connection header received
0928 08:56:13.896183 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:13.896185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:13.896247 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:13.896400 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:13.897495 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:13.898371 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:13.899523 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:13.900039 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:13.900139 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:13.902575 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:13.902579 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:13.902579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:13.902590 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:13.904191 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:13.904641 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:13.904643 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:13.904644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:13.904735 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:13.904994 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:13.904994 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:13.904994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:13.905248 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:13.905309 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:13.905669 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 08:56:13.905771 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":57,"threads_started":1}
I20260928 08:56:13.906616 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:13.895303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:34325 (local address 127.23.167.193:57457)
0928 08:56:13.895446 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:13.895455 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:13.895508 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:13.896466 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:13.896469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:13.899117 (+ 2648us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:56:13.899202 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:13.899205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:13.900387 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:13.900389 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:13.901033 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:13.901035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:13.901696 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:13.901697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:13.901750 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:13.902137 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:56:13.902142 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:56:13.903575 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:56:13.905426 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:13.905427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:13.905427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:13.905589 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:13.905837 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:13.905838 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:13.905838 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:13.905904 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:13.906311 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:56:13.906312 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:56:13.906357 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 08:56:13.906424 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 08:56:13.906517 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":17,"thread_start_us":66,"threads_started":1}
I20260928 08:56:13.906790 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:13.895419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:57457 (local address 127.23.167.195:34325)
0928 08:56:13.895560 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:13.895566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:13.895591 (+ 25us) server_negotiation.cc:407] Connection header received
0928 08:56:13.896181 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:13.896185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:13.896247 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:13.896399 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:13.899296 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:13.900317 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:13.901133 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:13.901624 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:13.901735 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:13.903673 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:13.903676 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:13.903676 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:13.903686 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:13.905352 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:13.905665 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:13.905667 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:13.905668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:13.905768 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:13.905980 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:13.905981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:13.905981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:13.906211 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:13.906281 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:13.906497 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 08:56:13.906599 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":55,"threads_started":1}
W20260928 08:56:13.906852 6663 leader_election.cc:341] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 00f838a4f4cc497bacd133ba8a2999b3 (127.23.167.194:34305): Not found: Tablet not found: 1b1219c3c1e642229dcc0a0a9374b817
W20260928 08:56:13.907508 6662 leader_election.cc:341] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f9e21caa9434b1f87809d96b2d26cfa (127.23.167.195:34325): Not found: Tablet not found: 1b1219c3c1e642229dcc0a0a9374b817
I20260928 08:56:13.907564 6662 leader_election.cc:302] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d135091233e4b609b1a7a67b7e5278f; no voters: 00f838a4f4cc497bacd133ba8a2999b3, 1f9e21caa9434b1f87809d96b2d26cfa
I20260928 08:56:13.907691 7050 raft_consensus.cc:2726] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:56:14.551126 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:14.550903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.193:37353)
0928 08:56:14.551035 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:14.551076 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260928 08:56:14.615101 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:14.614876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.194:55767)
0928 08:56:14.615023 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:14.615061 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":64,"threads_started":1}
I20260928 08:56:14.825026 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:14.824784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.195:39887)
0928 08:56:14.824937 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:14.824984 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":61,"threads_started":1}
I20260928 08:56:15.551769 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:15.551536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.193:56829)
0928 08:56:15.551677 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:15.551720 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":61,"threads_started":1}
I20260928 08:56:15.615882 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:15.615635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.194:53683)
0928 08:56:15.615778 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:15.615835 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":57,"threads_started":1}
I20260928 08:56:15.825677 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:15.825489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.195:33839)
0928 08:56:15.825600 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:15.825636 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":61,"threads_started":1}
I20260928 08:56:15.825753 7060 raft_consensus.cc:482] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:56:15.825841 7060 raft_consensus.cc:504] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 } } peers { permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 } } peers { permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 } }
I20260928 08:56:15.825973 7060 leader_election.cc:288] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f9e21caa9434b1f87809d96b2d26cfa (127.23.167.195:34325), 00f838a4f4cc497bacd133ba8a2999b3 (127.23.167.194:34305)
W20260928 08:56:15.826305 6662 leader_election.cc:341] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f9e21caa9434b1f87809d96b2d26cfa (127.23.167.195:34325): Not found: Tablet not found: 1b1219c3c1e642229dcc0a0a9374b817
W20260928 08:56:15.826367 6663 leader_election.cc:341] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 00f838a4f4cc497bacd133ba8a2999b3 (127.23.167.194:34305): Not found: Tablet not found: 1b1219c3c1e642229dcc0a0a9374b817
I20260928 08:56:15.826443 6662 leader_election.cc:302] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d135091233e4b609b1a7a67b7e5278f; no voters: 00f838a4f4cc497bacd133ba8a2999b3, 1f9e21caa9434b1f87809d96b2d26cfa
I20260928 08:56:15.826514 7060 raft_consensus.cc:2726] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:56:16.552480 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:16.552216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.193:47703)
0928 08:56:16.552367 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:16.552425 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":63,"threads_started":1}
I20260928 08:56:16.616607 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:16.616344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.194:58305)
0928 08:56:16.616487 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:16.616544 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260928 08:56:16.826256 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:16.826061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.195:39199)
0928 08:56:16.826184 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:16.826219 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
I20260928 08:56:17.553260 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:17.552992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.193:50183)
0928 08:56:17.553171 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:17.553213 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":121,"threads_started":1}
I20260928 08:56:17.617404 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:17.617168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.194:54483)
0928 08:56:17.617309 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:17.617357 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":58,"threads_started":1}
I20260928 08:56:17.763643 7066 raft_consensus.cc:482] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:56:17.763737 7066 raft_consensus.cc:504] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 } } peers { permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 } } peers { permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 } }
I20260928 08:56:17.763875 7066 leader_election.cc:288] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f9e21caa9434b1f87809d96b2d26cfa (127.23.167.195:34325), 00f838a4f4cc497bacd133ba8a2999b3 (127.23.167.194:34305)
W20260928 08:56:17.764241 6662 leader_election.cc:341] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f9e21caa9434b1f87809d96b2d26cfa (127.23.167.195:34325): Not found: Tablet not found: 1b1219c3c1e642229dcc0a0a9374b817
W20260928 08:56:17.764331 6663 leader_election.cc:341] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 00f838a4f4cc497bacd133ba8a2999b3 (127.23.167.194:34305): Not found: Tablet not found: 1b1219c3c1e642229dcc0a0a9374b817
I20260928 08:56:17.764379 6663 leader_election.cc:302] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d135091233e4b609b1a7a67b7e5278f; no voters: 00f838a4f4cc497bacd133ba8a2999b3, 1f9e21caa9434b1f87809d96b2d26cfa
I20260928 08:56:17.764437 7066 raft_consensus.cc:2726] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:56:17.826905 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:17.826671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.195:45661)
0928 08:56:17.826817 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:17.826860 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":60,"threads_started":1}
I20260928 08:56:18.553979 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:18.553736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.193:46261)
0928 08:56:18.553886 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:18.553930 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":77,"threads_started":1}
I20260928 08:56:18.618127 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:18.617893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.194:45461)
0928 08:56:18.618040 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:18.618083 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":61,"threads_started":1}
I20260928 08:56:18.827669 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:18.827421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.195:34373)
0928 08:56:18.827578 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:18.827624 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":78,"threads_started":1}
I20260928 08:56:19.554690 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:19.554417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.193:40207)
0928 08:56:19.554601 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:19.554644 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":77,"threads_started":1}
I20260928 08:56:19.618827 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:19.618596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.194:44203)
0928 08:56:19.618733 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:19.618780 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":59,"threads_started":1}
I20260928 08:56:19.828357 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:19.828152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.195:37207)
0928 08:56:19.828274 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:19.828314 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260928 08:56:20.318894 7074 raft_consensus.cc:482] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:56:20.318989 7074 raft_consensus.cc:504] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 } } peers { permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 } } peers { permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 } }
I20260928 08:56:20.319134 7074 leader_election.cc:288] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f9e21caa9434b1f87809d96b2d26cfa (127.23.167.195:34325), 00f838a4f4cc497bacd133ba8a2999b3 (127.23.167.194:34305)
W20260928 08:56:20.319496 6662 leader_election.cc:341] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f9e21caa9434b1f87809d96b2d26cfa (127.23.167.195:34325): Not found: Tablet not found: 1b1219c3c1e642229dcc0a0a9374b817
W20260928 08:56:20.319592 6663 leader_election.cc:341] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 00f838a4f4cc497bacd133ba8a2999b3 (127.23.167.194:34305): Not found: Tablet not found: 1b1219c3c1e642229dcc0a0a9374b817
I20260928 08:56:20.319641 6663 leader_election.cc:302] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d135091233e4b609b1a7a67b7e5278f; no voters: 00f838a4f4cc497bacd133ba8a2999b3, 1f9e21caa9434b1f87809d96b2d26cfa
I20260928 08:56:20.319721 7074 raft_consensus.cc:2726] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:56:20.555435 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:20.555202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.193:37695)
0928 08:56:20.555343 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:20.555388 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20260928 08:56:20.619603 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:20.619322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.194:57189)
0928 08:56:20.619509 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:20.619556 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":78,"threads_started":1}
I20260928 08:56:20.829016 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:20.828796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.195:43265)
0928 08:56:20.828927 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:20.828971 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":59,"threads_started":1}
I20260928 08:56:21.556134 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:21.555883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.193:58025)
0928 08:56:21.556038 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:21.556085 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":66,"threads_started":1}
I20260928 08:56:21.620352 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:21.620104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.194:57171)
0928 08:56:21.620259 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:21.620306 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":62,"threads_started":1}
I20260928 08:56:21.829747 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:21.829522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.195:56939)
0928 08:56:21.829665 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:21.829700 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
I20260928 08:56:22.556811 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:22.556565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.193:56991)
0928 08:56:22.556723 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:22.556762 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
I20260928 08:56:22.621160 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:22.620865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.194:40199)
0928 08:56:22.621040 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:22.621103 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":70,"threads_started":1}
I20260928 08:56:22.776768 7083 raft_consensus.cc:482] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:56:22.776868 7083 raft_consensus.cc:504] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 } } peers { permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 } } peers { permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 } }
I20260928 08:56:22.777038 7083 leader_election.cc:288] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f9e21caa9434b1f87809d96b2d26cfa (127.23.167.195:34325), 00f838a4f4cc497bacd133ba8a2999b3 (127.23.167.194:34305)
W20260928 08:56:22.777451 6663 leader_election.cc:341] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 00f838a4f4cc497bacd133ba8a2999b3 (127.23.167.194:34305): Not found: Tablet not found: 1b1219c3c1e642229dcc0a0a9374b817
W20260928 08:56:22.777603 6662 leader_election.cc:341] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f9e21caa9434b1f87809d96b2d26cfa (127.23.167.195:34325): Not found: Tablet not found: 1b1219c3c1e642229dcc0a0a9374b817
I20260928 08:56:22.777663 6662 leader_election.cc:302] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d135091233e4b609b1a7a67b7e5278f; no voters: 00f838a4f4cc497bacd133ba8a2999b3, 1f9e21caa9434b1f87809d96b2d26cfa
I20260928 08:56:22.777745 7083 raft_consensus.cc:2726] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:56:22.830421 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:22.830210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.195:40991)
0928 08:56:22.830335 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:22.830375 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260928 08:56:23.557479 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:23.557253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.193:40593)
0928 08:56:23.557395 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:23.557433 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260928 08:56:23.621826 7086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:23.621586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.194:52199)
0928 08:56:23.621742 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:23.621783 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":75,"threads_started":1}
I20260928 08:56:23.831063 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:23.830844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.195:52811)
0928 08:56:23.830981 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:23.831020 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260928 08:56:24.558064 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:24.557860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.193:44015)
0928 08:56:24.557983 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:24.558020 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":75,"threads_started":1}
I20260928 08:56:24.622582 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:24.622333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.194:54093)
0928 08:56:24.622492 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:24.622533 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":86,"threads_started":1}
I20260928 08:56:24.831802 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:24.831581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.195:51097)
0928 08:56:24.831716 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:24.831752 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
I20260928 08:56:25.558739 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:25.558462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.193:35851)
0928 08:56:25.558646 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:25.558690 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":116,"threads_started":1}
I20260928 08:56:25.623365 7092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:25.623099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.194:50143)
0928 08:56:25.623268 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:25.623317 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":98,"threads_started":1}
I20260928 08:56:25.832598 7093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:25.832332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.195:43709)
0928 08:56:25.832500 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:25.832550 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":77,"threads_started":1}
I20260928 08:56:26.559371 7094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:26.559164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.193:34567)
0928 08:56:26.559283 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:26.559324 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":66,"threads_started":1}
I20260928 08:56:26.624033 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:26.623828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.194:57711)
0928 08:56:26.623946 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:26.623986 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260928 08:56:26.833349 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:26.833115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.195:46965)
0928 08:56:26.833246 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:26.833294 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":58,"threads_started":1}
I20260928 08:56:27.426702 7097 raft_consensus.cc:482] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:56:27.426810 7097 raft_consensus.cc:504] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9e21caa9434b1f87809d96b2d26cfa" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 34325 } } peers { permanent_uuid: "00f838a4f4cc497bacd133ba8a2999b3" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34305 } } peers { permanent_uuid: "6d135091233e4b609b1a7a67b7e5278f" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 34943 } }
I20260928 08:56:27.426964 7097 leader_election.cc:288] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f9e21caa9434b1f87809d96b2d26cfa (127.23.167.195:34325), 00f838a4f4cc497bacd133ba8a2999b3 (127.23.167.194:34305)
W20260928 08:56:27.427377 6662 leader_election.cc:341] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f9e21caa9434b1f87809d96b2d26cfa (127.23.167.195:34325): Not found: Tablet not found: 1b1219c3c1e642229dcc0a0a9374b817
W20260928 08:56:27.427462 6663 leader_election.cc:341] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 00f838a4f4cc497bacd133ba8a2999b3 (127.23.167.194:34305): Not found: Tablet not found: 1b1219c3c1e642229dcc0a0a9374b817
I20260928 08:56:27.427521 6663 leader_election.cc:302] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d135091233e4b609b1a7a67b7e5278f; no voters: 00f838a4f4cc497bacd133ba8a2999b3, 1f9e21caa9434b1f87809d96b2d26cfa
I20260928 08:56:27.427587 7097 raft_consensus.cc:2726] T 1b1219c3c1e642229dcc0a0a9374b817 P 6d135091233e4b609b1a7a67b7e5278f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:56:27.560228 7098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:27.559964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.193:46495)
0928 08:56:27.560139 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:27.560182 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":85,"threads_started":1}
I20260928 08:56:27.624817 7099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:27.624558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.194:46681)
0928 08:56:27.624719 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:27.624770 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":65,"threads_started":1}
I20260928 08:56:27.834201 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:27.833960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:42107 (local address 127.23.167.195:34865)
0928 08:56:27.834102 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:27.834142 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260928 08:56:27.841928 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 6617
I20260928 08:56:27.846213 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 6781
I20260928 08:56:27.850005 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 6914
2026-09-28T08:56:27Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18796 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:56:28 dist-test-slave-pbjp krb5kdc[7103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:56:28 dist-test-slave-pbjp krb5kdc[7103](info): set up 2 sockets
Sep 28 08:56:28 dist-test-slave-pbjp krb5kdc[7103](info): commencing operation
krb5kdc: starting...
W20260928 08:56:29.068969 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:56:29 dist-test-slave-pbjp krb5kdc[7103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:56:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:56:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.167.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
I20260928 08:56:29.182982 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:36677
--webserver_interface=127.23.167.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:36677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:56:29.261812 7116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:56:29.262022 7116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:56:29.262042 7116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:56:29.263131 7116 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 08:56:29.263159 7116 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 08:56:29.263171 7116 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 08:56:29.263180 7116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:56:29.263192 7116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:56:29.263203 7116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:56:29.263213 7116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:56:29.263224 7116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:56:29.264622 7116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:36677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:36677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:56:29.264809 7116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:56:29.264992 7116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:56:29.270155 7121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:29.270253 7122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:29.270159 7124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:29.270275 7116 server_base.cc:1034] running on GCE node
I20260928 08:56:29.270750 7116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:56:29.271065 7116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:56:29.272233 7116 hybrid_clock.cc:648] HybridClock initialized: now 1790585789272216 us; error 29 us; skew 500 ppm
Sep 28 08:56:29 dist-test-slave-pbjp krb5kdc[7103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585789, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:29.274909 7116 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:56:29.275343 7116 webserver.cc:460] Webserver started at http://127.23.167.254:37549/ using document root <none> and password file <none>
I20260928 08:56:29.275615 7116 fs_manager.cc:359] Metadata directory not provided
I20260928 08:56:29.275669 7116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:56:29.275792 7116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:56:29.276841 7116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "c6086acd2a1844d0822305982e68b0f5"
format_stamp: "Formatted at 2026-09-28 08:56:29 on dist-test-slave-pbjp"
server_key: "7b864a89a1e9e7eb8a05eeeac8d2daac"
server_key_iv: "f3e5c995f9e8e3af6be6421857091dbe"
server_key_version: "encryptionkey@0"
I20260928 08:56:29.277194 7116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "c6086acd2a1844d0822305982e68b0f5"
format_stamp: "Formatted at 2026-09-28 08:56:29 on dist-test-slave-pbjp"
server_key: "7b864a89a1e9e7eb8a05eeeac8d2daac"
server_key_iv: "f3e5c995f9e8e3af6be6421857091dbe"
server_key_version: "encryptionkey@0"
I20260928 08:56:29.278461 7116 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:56:29.279250 7131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:29.279448 7116 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:56:29.279551 7116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "c6086acd2a1844d0822305982e68b0f5"
format_stamp: "Formatted at 2026-09-28 08:56:29 on dist-test-slave-pbjp"
server_key: "7b864a89a1e9e7eb8a05eeeac8d2daac"
server_key_iv: "f3e5c995f9e8e3af6be6421857091dbe"
server_key_version: "encryptionkey@0"
I20260928 08:56:29.279615 7116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:56:29.296032 7116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:56:29.298177 7116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:56:29.298311 7116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:56:29.302817 7116 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:36677
I20260928 08:56:29.302856 7183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:36677 every 8 connection(s)
I20260928 08:56:29.303301 7116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:56:29.304603 7184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:56:29.307353 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 7116
I20260928 08:56:29.307443 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:56:29.307636 24223 external_mini_cluster.cc:1428] Setting key 51ac60a38bc3cdc1a02fc4c0e2f8f086
I20260928 08:56:29.308320 7184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5: Bootstrap starting.
I20260928 08:56:29.308928 7184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5: Neither blocks nor log segments found. Creating new log.
I20260928 08:56:29.309255 7184 log.cc:824] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5: Log is configured to *not* fsync() on all Append() calls
I20260928 08:56:29.309978 7184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5: No bootstrap required, opened a new log
I20260928 08:56:29.311724 7184 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6086acd2a1844d0822305982e68b0f5" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 36677 } }
I20260928 08:56:29.311865 7184 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:56:29.311896 7184 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6086acd2a1844d0822305982e68b0f5, State: Initialized, Role: FOLLOWER
I20260928 08:56:29.312041 7184 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6086acd2a1844d0822305982e68b0f5" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 36677 } }
I20260928 08:56:29.312119 7184 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:56:29.312150 7184 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:56:29.312198 7184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:56:29.312799 7184 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6086acd2a1844d0822305982e68b0f5" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 36677 } }
I20260928 08:56:29.312901 7184 leader_election.cc:302] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6086acd2a1844d0822305982e68b0f5; no voters:
I20260928 08:56:29.313050 7184 leader_election.cc:288] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:56:29.313150 7189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:56:29.313354 7184 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:56:29.313532 7189 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [term 1 LEADER]: Becoming Leader. State: Replica: c6086acd2a1844d0822305982e68b0f5, State: Running, Role: LEADER
I20260928 08:56:29.313692 7189 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6086acd2a1844d0822305982e68b0f5" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 36677 } }
Sep 28 08:56:29 dist-test-slave-pbjp krb5kdc[7103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:56:29.314069 7190 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6086acd2a1844d0822305982e68b0f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6086acd2a1844d0822305982e68b0f5" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 36677 } } }
I20260928 08:56:29.314137 7190 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [sys.catalog]: This master's current role is: LEADER
I20260928 08:56:29.314285 7191 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6086acd2a1844d0822305982e68b0f5. Latest consensus state: current_term: 1 leader_uuid: "c6086acd2a1844d0822305982e68b0f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6086acd2a1844d0822305982e68b0f5" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 36677 } } }
I20260928 08:56:29.314339 7191 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [sys.catalog]: This master's current role is: LEADER
I20260928 08:56:29.314720 7198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:56:29.315276 7198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:56:29.316939 7198 catalog_manager.cc:1347] Generated new cluster ID: 6103a9ef8f24456697e39fc3776df251
I20260928 08:56:29.316998 7198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:56:29.318337 7187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:29.308156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36588 (local address 127.23.167.254:36677)
0928 08:56:29.308355 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:29.308365 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:29.308391 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:56:29.308824 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:29.308830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:29.308878 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:29.309007 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:29.309848 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.310721 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.311437 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.311854 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.311948 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:29.314731 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:29.314735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:29.314736 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:29.314748 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:29.316702 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:29.317136 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:29.317139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:29.317140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:29.317241 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:29.317452 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:29.317453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:29.317453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:29.317734 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:29.317862 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:29.317998 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 08:56:29.318130 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
I20260928 08:56:29.324800 7198 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:56:29.325344 7198 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:56:29.333940 7198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5: Generated new TSK 0
I20260928 08:56:29.334115 7198 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.167.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
I20260928 08:56:29.359516 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:0
--local_ip_for_outbound_sockets=127.23.167.193
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:36677
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:56:29.439574 7210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:56:29.439783 7210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:56:29.439805 7210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:56:29.440905 7210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:56:29.440939 7210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:56:29.440963 7210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:56:29.442373 7210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:36677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:56:29.442554 7210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:56:29.442741 7210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:56:29.447419 7216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:29.447441 7215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:29.447566 7218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:29.447569 7210 server_base.cc:1034] running on GCE node
I20260928 08:56:29.447872 7210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:56:29.448096 7210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:56:29.449229 7210 hybrid_clock.cc:648] HybridClock initialized: now 1790585789449216 us; error 29 us; skew 500 ppm
Sep 28 08:56:29 dist-test-slave-pbjp krb5kdc[7103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585789, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:29.451562 7210 init.cc:377] Logged in from keytab as kudu/127.23.167.193@KRBTEST.COM (short username kudu)
I20260928 08:56:29.451982 7210 webserver.cc:460] Webserver started at http://127.23.167.193:41123/ using document root <none> and password file <none>
I20260928 08:56:29.452199 7210 fs_manager.cc:359] Metadata directory not provided
I20260928 08:56:29.452248 7210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:56:29.452368 7210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:56:29.453459 7210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/data/instance:
uuid: "e2c63387905b4bef8a4057433fedefca"
format_stamp: "Formatted at 2026-09-28 08:56:29 on dist-test-slave-pbjp"
server_key: "80cf625c1f62990d4b2bedfba10fc64f"
server_key_iv: "1ad0219a8e892db7fc78781f397d7fb3"
server_key_version: "encryptionkey@0"
I20260928 08:56:29.453784 7210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance:
uuid: "e2c63387905b4bef8a4057433fedefca"
format_stamp: "Formatted at 2026-09-28 08:56:29 on dist-test-slave-pbjp"
server_key: "80cf625c1f62990d4b2bedfba10fc64f"
server_key_iv: "1ad0219a8e892db7fc78781f397d7fb3"
server_key_version: "encryptionkey@0"
I20260928 08:56:29.454962 7210 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:56:29.455639 7225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:29.455775 7210 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:56:29.455856 7210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "e2c63387905b4bef8a4057433fedefca"
format_stamp: "Formatted at 2026-09-28 08:56:29 on dist-test-slave-pbjp"
server_key: "80cf625c1f62990d4b2bedfba10fc64f"
server_key_iv: "1ad0219a8e892db7fc78781f397d7fb3"
server_key_version: "encryptionkey@0"
I20260928 08:56:29.455916 7210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:56:29.465524 7210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:56:29.467319 7210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:56:29.467450 7210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:56:29.467665 7210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:56:29.467954 7210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:56:29.467986 7210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:29.468016 7210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:56:29.468039 7210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:29.473516 7210 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:46519
I20260928 08:56:29.473623 7338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:46519 every 8 connection(s)
I20260928 08:56:29.473883 7210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
Sep 28 08:56:29 dist-test-slave-pbjp krb5kdc[7103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585789, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:56:29.482985 7187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:29.474648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:44257 (local address 127.23.167.254:36677)
0928 08:56:29.474720 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:29.474723 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:29.474881 (+ 158us) server_negotiation.cc:407] Connection header received
0928 08:56:29.475442 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:29.475443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:29.475457 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:29.475510 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:29.476504 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.477055 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.478070 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.478293 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.478352 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:29.480433 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:29.480435 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:29.480435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:29.480444 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:29.481592 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:29.481959 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:29.481960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:29.481961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:29.482022 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:29.482323 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:29.482324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:29.482324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:29.482447 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:29.482503 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:29.482791 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 08:56:29.482891 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 08:56:29.483194 7341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:29.474560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.193:44257)
0928 08:56:29.474749 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:29.474798 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:29.475335 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:29.475606 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:29.475610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:29.476194 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:56:29.476401 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:29.476406 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.477176 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.477177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:29.477980 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:29.477982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.478368 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.478369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:29.478494 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:29.478947 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:56:29.478956 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:56:29.480339 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:56:29.481667 (+ 1328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:29.481675 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:29.481677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:29.481893 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:29.482077 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:29.482078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:29.482078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:29.482255 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:29.482502 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:56:29.482510 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:56:29.482655 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 08:56:29.482759 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 08:56:29.482927 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":99,"threads_started":1}
I20260928 08:56:29.483752 7339 heartbeater.cc:347] Connected to a master server at 127.23.167.254:36677
I20260928 08:56:29.483863 7339 heartbeater.cc:464] Registering TS with master...
I20260928 08:56:29.484009 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 7210
I20260928 08:56:29.484057 7339 heartbeater.cc:511] Master 127.23.167.254:36677 requested a full tablet report, sending...
I20260928 08:56:29.484097 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance
I20260928 08:56:29.484310 24223 external_mini_cluster.cc:1428] Setting key aae548763548b3276101c7d18b25ec65
I20260928 08:56:29.484690 7147 ts_manager.cc:194] Registered new tserver with Master: e2c63387905b4bef8a4057433fedefca (127.23.167.193:46519)
I20260928 08:56:29.485419 7147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.193@KRBTEST.COM'} at 127.23.167.193:44257
WARNING: no policy specified for kudu/127.23.167.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
I20260928 08:56:29.508404 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:0
--local_ip_for_outbound_sockets=127.23.167.194
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:36677
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:56:29.588230 7344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:56:29.588418 7344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:56:29.588447 7344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:56:29.589564 7344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:56:29.589601 7344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:56:29.589635 7344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:56:29.591009 7344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:36677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:56:29.591236 7344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:56:29.591429 7344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 08:56:29.596009 7344 server_base.cc:1034] running on GCE node
W20260928 08:56:29.596060 7349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:29.595988 7350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:29.595994 7352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:29.596426 7344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:56:29.596657 7344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:56:29.597801 7344 hybrid_clock.cc:648] HybridClock initialized: now 1790585789597784 us; error 31 us; skew 500 ppm
Sep 28 08:56:29 dist-test-slave-pbjp krb5kdc[7103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585789, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:29.600073 7344 init.cc:377] Logged in from keytab as kudu/127.23.167.194@KRBTEST.COM (short username kudu)
I20260928 08:56:29.600447 7344 webserver.cc:460] Webserver started at http://127.23.167.194:41501/ using document root <none> and password file <none>
I20260928 08:56:29.600636 7344 fs_manager.cc:359] Metadata directory not provided
I20260928 08:56:29.600675 7344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:56:29.600762 7344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:56:29.601744 7344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/data/instance:
uuid: "ffb3b609b7154fa4937dd34afe6adf7b"
format_stamp: "Formatted at 2026-09-28 08:56:29 on dist-test-slave-pbjp"
server_key: "1ee48a1a821ea765f9b6ceddbe636281"
server_key_iv: "86562fe4997eb5285122c99efe4b06f9"
server_key_version: "encryptionkey@0"
I20260928 08:56:29.602026 7344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance:
uuid: "ffb3b609b7154fa4937dd34afe6adf7b"
format_stamp: "Formatted at 2026-09-28 08:56:29 on dist-test-slave-pbjp"
server_key: "1ee48a1a821ea765f9b6ceddbe636281"
server_key_iv: "86562fe4997eb5285122c99efe4b06f9"
server_key_version: "encryptionkey@0"
I20260928 08:56:29.603188 7344 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:56:29.603895 7359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:29.604033 7344 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:56:29.604104 7344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "ffb3b609b7154fa4937dd34afe6adf7b"
format_stamp: "Formatted at 2026-09-28 08:56:29 on dist-test-slave-pbjp"
server_key: "1ee48a1a821ea765f9b6ceddbe636281"
server_key_iv: "86562fe4997eb5285122c99efe4b06f9"
server_key_version: "encryptionkey@0"
I20260928 08:56:29.604169 7344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:56:29.620693 7344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:56:29.623011 7344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:56:29.623135 7344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:56:29.623368 7344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:56:29.623684 7344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:56:29.623718 7344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:29.623749 7344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:56:29.623767 7344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:29.628975 7344 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:44641
I20260928 08:56:29.629035 7472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:44641 every 8 connection(s)
I20260928 08:56:29.629359 7344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
I20260928 08:56:29.633013 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 7344
I20260928 08:56:29.633157 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance
I20260928 08:56:29.633354 24223 external_mini_cluster.cc:1428] Setting key 34cea030a8348d4fd39ce4f7944948ab
Sep 28 08:56:29 dist-test-slave-pbjp krb5kdc[7103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585789, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:56:29.638829 7187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:29.630076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:50277 (local address 127.23.167.254:36677)
0928 08:56:29.630136 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:29.630138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:29.630344 (+ 206us) server_negotiation.cc:407] Connection header received
0928 08:56:29.630827 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:29.630828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:29.630837 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:29.630889 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:29.631789 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.632282 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.633565 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.633811 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.633864 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:29.636031 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:29.636032 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:29.636032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:29.636040 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:29.637352 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:29.637750 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:29.637751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:29.637751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:29.637820 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:29.638109 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:29.638110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:29.638110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:29.638253 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:29.638325 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:29.638628 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 08:56:29.638728 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 08:56:29.638985 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:29.630036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.194:50277)
0928 08:56:29.630221 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:29.630264 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:29.630752 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:29.630952 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:29.630956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:29.631518 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:56:29.631723 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:29.631728 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.632366 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.632367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:29.633455 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:29.633458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.633883 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.633884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:29.634020 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:29.634514 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:56:29.634522 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:56:29.635935 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:56:29.637425 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:29.637431 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:29.637433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:29.637686 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:29.637913 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:29.637914 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:29.637914 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:29.638035 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:29.638314 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:56:29.638317 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:56:29.638489 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 08:56:29.638579 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 08:56:29.638718 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":93,"threads_started":1}
I20260928 08:56:29.639518 7473 heartbeater.cc:347] Connected to a master server at 127.23.167.254:36677
I20260928 08:56:29.639617 7473 heartbeater.cc:464] Registering TS with master...
I20260928 08:56:29.639796 7473 heartbeater.cc:511] Master 127.23.167.254:36677 requested a full tablet report, sending...
I20260928 08:56:29.640180 7147 ts_manager.cc:194] Registered new tserver with Master: ffb3b609b7154fa4937dd34afe6adf7b (127.23.167.194:44641)
I20260928 08:56:29.640561 7147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.194@KRBTEST.COM'} at 127.23.167.194:50277
WARNING: no policy specified for kudu/127.23.167.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
I20260928 08:56:29.658205 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:0
--local_ip_for_outbound_sockets=127.23.167.195
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:36677
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:56:29.738952 7478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:56:29.739166 7478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:56:29.739198 7478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:56:29.740357 7478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:56:29.740401 7478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:56:29.740435 7478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:56:29.741896 7478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:36677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:56:29.742136 7478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:56:29.742352 7478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:56:29.747102 7486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:29.747152 7478 server_base.cc:1034] running on GCE node
W20260928 08:56:29.747082 7483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:29.747074 7484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:29.747447 7478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:56:29.747653 7478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:56:29.748798 7478 hybrid_clock.cc:648] HybridClock initialized: now 1790585789748787 us; error 28 us; skew 500 ppm
Sep 28 08:56:29 dist-test-slave-pbjp krb5kdc[7103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585789, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:29.751173 7478 init.cc:377] Logged in from keytab as kudu/127.23.167.195@KRBTEST.COM (short username kudu)
I20260928 08:56:29.751554 7478 webserver.cc:460] Webserver started at http://127.23.167.195:45705/ using document root <none> and password file <none>
I20260928 08:56:29.751750 7478 fs_manager.cc:359] Metadata directory not provided
I20260928 08:56:29.751797 7478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:56:29.751915 7478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:56:29.752948 7478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/data/instance:
uuid: "a6b8d92ca2404ddeb0a1a52edbae0575"
format_stamp: "Formatted at 2026-09-28 08:56:29 on dist-test-slave-pbjp"
server_key: "c81c39ea847e05682a2f036b23a46744"
server_key_iv: "a9ffed92561b9e94d80b335377750023"
server_key_version: "encryptionkey@0"
I20260928 08:56:29.753324 7478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance:
uuid: "a6b8d92ca2404ddeb0a1a52edbae0575"
format_stamp: "Formatted at 2026-09-28 08:56:29 on dist-test-slave-pbjp"
server_key: "c81c39ea847e05682a2f036b23a46744"
server_key_iv: "a9ffed92561b9e94d80b335377750023"
server_key_version: "encryptionkey@0"
I20260928 08:56:29.754585 7478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:56:29.755312 7493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:29.755476 7478 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:29.755555 7478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "a6b8d92ca2404ddeb0a1a52edbae0575"
format_stamp: "Formatted at 2026-09-28 08:56:29 on dist-test-slave-pbjp"
server_key: "c81c39ea847e05682a2f036b23a46744"
server_key_iv: "a9ffed92561b9e94d80b335377750023"
server_key_version: "encryptionkey@0"
I20260928 08:56:29.755620 7478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:56:29.780767 7478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:56:29.783017 7478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:56:29.783182 7478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:56:29.783427 7478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:56:29.783769 7478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:56:29.783807 7478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:29.783830 7478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:56:29.783841 7478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:29.789335 7478 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:36981
I20260928 08:56:29.789405 7606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:36981 every 8 connection(s)
I20260928 08:56:29.789695 7478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
I20260928 08:56:29.792974 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 7478
I20260928 08:56:29.793083 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance
I20260928 08:56:29.793282 24223 external_mini_cluster.cc:1428] Setting key e23613c0ae542f4200052941098e4d6e
Sep 28 08:56:29 dist-test-slave-pbjp krb5kdc[7103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585789, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:56:29.798444 7187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:29.790378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:38317 (local address 127.23.167.254:36677)
0928 08:56:29.790428 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:29.790430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:29.790648 (+ 218us) server_negotiation.cc:407] Connection header received
0928 08:56:29.791148 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:29.791149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:29.791158 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:29.791200 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:29.792137 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.792626 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.793896 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.794159 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.794220 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:29.796184 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:29.796185 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:29.796186 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:29.796194 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:29.797266 (+ 1072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:29.797610 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:29.797611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:29.797611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:29.797668 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:29.797874 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:29.797875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:29.797875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:29.797981 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:29.798036 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:29.798284 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 08:56:29.798358 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 08:56:29.798625 7609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:29.790352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.195:38317)
0928 08:56:29.790520 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:29.790570 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:29.791038 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:29.791273 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:29.791276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:29.791872 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:56:29.792069 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:29.792074 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.792702 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.792703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:29.793824 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:29.793827 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.794244 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.794245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:29.794348 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:29.794818 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:56:29.794827 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:56:29.796098 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:56:29.797338 (+ 1240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:29.797344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:29.797347 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:29.797547 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:29.797719 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:29.797720 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:29.797720 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:29.797820 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:29.798033 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:56:29.798039 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:56:29.798171 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 08:56:29.798259 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 08:56:29.798408 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
I20260928 08:56:29.799008 7607 heartbeater.cc:347] Connected to a master server at 127.23.167.254:36677
I20260928 08:56:29.799094 7607 heartbeater.cc:464] Registering TS with master...
I20260928 08:56:29.799273 7607 heartbeater.cc:511] Master 127.23.167.254:36677 requested a full tablet report, sending...
I20260928 08:56:29.799640 7147 ts_manager.cc:194] Registered new tserver with Master: a6b8d92ca2404ddeb0a1a52edbae0575 (127.23.167.195:36981)
I20260928 08:56:29.799979 7147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.195@KRBTEST.COM'} at 127.23.167.195:38317
I20260928 08:56:29.805047 24223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:56:29.811714 7187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:29.806400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36596 (local address 127.23.167.254:36677)
0928 08:56:29.806480 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:29.806482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:29.806579 (+ 97us) server_negotiation.cc:407] Connection header received
0928 08:56:29.806642 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:29.806643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:29.806670 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:29.806722 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:29.807231 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.807712 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.808410 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.808602 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.808651 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:29.809744 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:29.809746 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:29.809746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:29.809752 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:29.810862 (+ 1110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:29.811149 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:29.811150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:29.811150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:29.811194 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:29.811320 (+ 126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:29.811320 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:29.811320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:29.811451 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:29.811524 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:29.811594 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 08:56:29.811648 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 08:56:29.813313 7147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:56:29.814970 7147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:56:29.820835 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:29.816921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:36981 (local address 127.0.0.1:49758)
0928 08:56:29.817381 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:29.817392 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:29.817442 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:29.817671 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:29.817672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:29.817674 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:56:29.817712 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:29.817713 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.818987 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.818988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:29.819802 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:29.819804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.820542 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.820542 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:29.820621 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:29.820636 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 08:56:29.820705 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 08:56:29.820755 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":1,"thread_start_us":50,"threads_started":1}
I20260928 08:56:29.820848 7617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:29.817021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49758 (local address 127.23.167.195:36981)
0928 08:56:29.817419 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:29.817422 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:29.817432 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:56:29.817513 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:29.817515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:29.817530 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:29.817607 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:56:29.817781 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.818871 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.819884 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.820464 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.820549 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:29.820603 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:29.820725 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 08:56:29.820767 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":116,"threads_started":1}
I20260928 08:56:29.821383 7620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:29.817224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58016 (local address 127.23.167.193:46519)
0928 08:56:29.817372 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:29.817378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:29.817641 (+ 263us) server_negotiation.cc:407] Connection header received
0928 08:56:29.817696 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:29.817699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:29.817715 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:29.817781 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:56:29.817964 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.818744 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.820189 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.821002 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.821109 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:29.821171 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:29.821263 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 08:56:29.821307 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
I20260928 08:56:29.821379 7615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:29.817120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:46519 (local address 127.0.0.1:58016)
0928 08:56:29.817521 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:29.817529 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:29.817569 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:29.817861 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:29.817862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:29.817864 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:56:29.817907 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:29.817909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.818831 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.818832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:29.820109 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:29.820112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.821123 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.821124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:29.821191 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:29.821196 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 08:56:29.821250 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 08:56:29.821289 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":61,"thread_start_us":39,"threads_started":1}
I20260928 08:56:29.821379 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:29.816908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:44641 (local address 127.0.0.1:45408)
0928 08:56:29.817080 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:29.817119 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:29.817187 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:29.817558 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:29.817561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:29.817567 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:56:29.817634 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:29.817636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.818975 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.818975 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:29.820320 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:29.820322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.821186 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.821187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:29.821237 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:29.821240 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:56:29.821267 (+ 27us) client_negotiation.cc:240] Negotiation successful
0928 08:56:29.821294 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":33,"thread_start_us":54,"threads_started":1}
I20260928 08:56:29.821748 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:29.817229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45408 (local address 127.23.167.194:44641)
0928 08:56:29.817381 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:29.817386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:29.817397 (+ 11us) server_negotiation.cc:407] Connection header received
0928 08:56:29.817439 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:29.817441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:29.817457 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:29.817534 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:56:29.817721 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.818877 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.820387 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.821103 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.821527 (+ 424us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:29.821592 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:29.821621 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 08:56:29.821660 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
I20260928 08:56:29.821724 7541 tablet_service.cc:1467] Processing CreateTablet for tablet 5c46142b4a184014a1cebcc6fb287e9d (DEFAULT_TABLE table=test-table [id=608b562228f64103a1401c3ba4867f40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:56:29.822116 7541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c46142b4a184014a1cebcc6fb287e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:56:29.822165 7273 tablet_service.cc:1467] Processing CreateTablet for tablet 5c46142b4a184014a1cebcc6fb287e9d (DEFAULT_TABLE table=test-table [id=608b562228f64103a1401c3ba4867f40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:56:29.822427 7273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c46142b4a184014a1cebcc6fb287e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:56:29.824661 7622 tablet_bootstrap.cc:492] T 5c46142b4a184014a1cebcc6fb287e9d P e2c63387905b4bef8a4057433fedefca: Bootstrap starting.
I20260928 08:56:29.824570 7621 tablet_bootstrap.cc:492] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575: Bootstrap starting.
I20260928 08:56:29.825363 7621 tablet_bootstrap.cc:654] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575: Neither blocks nor log segments found. Creating new log.
I20260928 08:56:29.825366 7622 tablet_bootstrap.cc:654] T 5c46142b4a184014a1cebcc6fb287e9d P e2c63387905b4bef8a4057433fedefca: Neither blocks nor log segments found. Creating new log.
I20260928 08:56:29.825357 7407 tablet_service.cc:1467] Processing CreateTablet for tablet 5c46142b4a184014a1cebcc6fb287e9d (DEFAULT_TABLE table=test-table [id=608b562228f64103a1401c3ba4867f40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:56:29.825695 7407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c46142b4a184014a1cebcc6fb287e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:56:29.825862 7621 log.cc:824] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575: Log is configured to *not* fsync() on all Append() calls
I20260928 08:56:29.825886 7622 log.cc:824] T 5c46142b4a184014a1cebcc6fb287e9d P e2c63387905b4bef8a4057433fedefca: Log is configured to *not* fsync() on all Append() calls
I20260928 08:56:29.826643 7621 tablet_bootstrap.cc:492] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575: No bootstrap required, opened a new log
I20260928 08:56:29.826718 7621 ts_tablet_manager.cc:1397] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 08:56:29.828764 7625 tablet_bootstrap.cc:492] T 5c46142b4a184014a1cebcc6fb287e9d P ffb3b609b7154fa4937dd34afe6adf7b: Bootstrap starting.
I20260928 08:56:29.828837 7621 raft_consensus.cc:348] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 44641 } } peers { permanent_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 36981 } } peers { permanent_uuid: "e2c63387905b4bef8a4057433fedefca" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46519 } }
I20260928 08:56:29.829008 7621 raft_consensus.cc:374] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:56:29.829035 7621 raft_consensus.cc:729] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6b8d92ca2404ddeb0a1a52edbae0575, State: Initialized, Role: FOLLOWER
I20260928 08:56:29.829182 7621 consensus_queue.cc:260] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 44641 } } peers { permanent_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 36981 } } peers { permanent_uuid: "e2c63387905b4bef8a4057433fedefca" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46519 } }
I20260928 08:56:29.829334 7625 tablet_bootstrap.cc:654] T 5c46142b4a184014a1cebcc6fb287e9d P ffb3b609b7154fa4937dd34afe6adf7b: Neither blocks nor log segments found. Creating new log.
I20260928 08:56:29.829444 7621 ts_tablet_manager.cc:1428] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 08:56:29.829468 7607 heartbeater.cc:503] Master 127.23.167.254:36677 was elected leader, sending a full tablet report...
I20260928 08:56:29.829834 7622 tablet_bootstrap.cc:492] T 5c46142b4a184014a1cebcc6fb287e9d P e2c63387905b4bef8a4057433fedefca: No bootstrap required, opened a new log
I20260928 08:56:29.829901 7622 ts_tablet_manager.cc:1397] T 5c46142b4a184014a1cebcc6fb287e9d P e2c63387905b4bef8a4057433fedefca: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 08:56:29.829913 7625 log.cc:824] T 5c46142b4a184014a1cebcc6fb287e9d P ffb3b609b7154fa4937dd34afe6adf7b: Log is configured to *not* fsync() on all Append() calls
I20260928 08:56:29.830600 7625 tablet_bootstrap.cc:492] T 5c46142b4a184014a1cebcc6fb287e9d P ffb3b609b7154fa4937dd34afe6adf7b: No bootstrap required, opened a new log
I20260928 08:56:29.830710 7625 ts_tablet_manager.cc:1397] T 5c46142b4a184014a1cebcc6fb287e9d P ffb3b609b7154fa4937dd34afe6adf7b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 08:56:29.831866 7622 raft_consensus.cc:348] T 5c46142b4a184014a1cebcc6fb287e9d P e2c63387905b4bef8a4057433fedefca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 44641 } } peers { permanent_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 36981 } } peers { permanent_uuid: "e2c63387905b4bef8a4057433fedefca" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46519 } }
I20260928 08:56:29.832044 7622 raft_consensus.cc:374] T 5c46142b4a184014a1cebcc6fb287e9d P e2c63387905b4bef8a4057433fedefca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:56:29.832077 7622 raft_consensus.cc:729] T 5c46142b4a184014a1cebcc6fb287e9d P e2c63387905b4bef8a4057433fedefca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c63387905b4bef8a4057433fedefca, State: Initialized, Role: FOLLOWER
I20260928 08:56:29.832196 7622 consensus_queue.cc:260] T 5c46142b4a184014a1cebcc6fb287e9d P e2c63387905b4bef8a4057433fedefca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 44641 } } peers { permanent_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 36981 } } peers { permanent_uuid: "e2c63387905b4bef8a4057433fedefca" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46519 } }
I20260928 08:56:29.832471 7622 ts_tablet_manager.cc:1428] T 5c46142b4a184014a1cebcc6fb287e9d P e2c63387905b4bef8a4057433fedefca: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:56:29.832489 7339 heartbeater.cc:503] Master 127.23.167.254:36677 was elected leader, sending a full tablet report...
I20260928 08:56:29.832541 7625 raft_consensus.cc:348] T 5c46142b4a184014a1cebcc6fb287e9d P ffb3b609b7154fa4937dd34afe6adf7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 44641 } } peers { permanent_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 36981 } } peers { permanent_uuid: "e2c63387905b4bef8a4057433fedefca" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46519 } }
I20260928 08:56:29.832690 7625 raft_consensus.cc:374] T 5c46142b4a184014a1cebcc6fb287e9d P ffb3b609b7154fa4937dd34afe6adf7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:56:29.832710 7625 raft_consensus.cc:729] T 5c46142b4a184014a1cebcc6fb287e9d P ffb3b609b7154fa4937dd34afe6adf7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffb3b609b7154fa4937dd34afe6adf7b, State: Initialized, Role: FOLLOWER
I20260928 08:56:29.832787 7625 consensus_queue.cc:260] T 5c46142b4a184014a1cebcc6fb287e9d P ffb3b609b7154fa4937dd34afe6adf7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 44641 } } peers { permanent_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 36981 } } peers { permanent_uuid: "e2c63387905b4bef8a4057433fedefca" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46519 } }
I20260928 08:56:29.833017 7625 ts_tablet_manager.cc:1428] T 5c46142b4a184014a1cebcc6fb287e9d P ffb3b609b7154fa4937dd34afe6adf7b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 08:56:29.833112 7473 heartbeater.cc:503] Master 127.23.167.254:36677 was elected leader, sending a full tablet report...
W20260928 08:56:29.879925 7474 tablet.cc:2369] T 5c46142b4a184014a1cebcc6fb287e9d P ffb3b609b7154fa4937dd34afe6adf7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:56:29.888195 7626 raft_consensus.cc:482] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:56:29.888324 7626 raft_consensus.cc:504] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 44641 } } peers { permanent_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 36981 } } peers { permanent_uuid: "e2c63387905b4bef8a4057433fedefca" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46519 } }
I20260928 08:56:29.888712 7626 leader_election.cc:288] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffb3b609b7154fa4937dd34afe6adf7b (127.23.167.194:44641), e2c63387905b4bef8a4057433fedefca (127.23.167.193:46519)
I20260928 08:56:29.891477 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:29.888931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:36659 (local address 127.23.167.194:44641)
0928 08:56:29.888991 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:29.888993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:29.889003 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:56:29.889051 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:29.889052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:29.889081 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:29.889138 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:56:29.889316 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.889823 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.890762 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.891161 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.891214 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:29.891221 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:29.891375 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 08:56:29.891408 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 08:56:29.891477 7609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:29.888871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:44641 (local address 127.23.167.195:36659)
0928 08:56:29.888938 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:29.888950 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:29.888998 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:29.889157 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:29.889158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:29.889160 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:56:29.889241 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:29.889242 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.889897 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.889898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:29.890676 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:29.890679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.891235 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.891236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:29.891310 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:29.891314 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:56:29.891364 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 08:56:29.891404 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 08:56:29.891546 7620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:29.889037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:42005 (local address 127.23.167.193:46519)
0928 08:56:29.889101 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:29.889104 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:29.889217 (+ 113us) server_negotiation.cc:407] Connection header received
0928 08:56:29.889249 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:29.889250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:29.889261 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:29.889312 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:56:29.889445 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.889929 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.890860 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.891244 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.891312 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:29.891319 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:29.891453 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 08:56:29.891487 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 08:56:29.891598 7630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:29.888964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:46519 (local address 127.23.167.195:42005)
0928 08:56:29.889151 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:29.889160 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:29.889208 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:29.889320 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:29.889321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:29.889322 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:56:29.889389 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:29.889390 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.890001 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.890002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:29.890772 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:29.890775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.891320 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.891321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:29.891383 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:29.891387 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:56:29.891434 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 08:56:29.891478 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":115,"threads_started":1}
I20260928 08:56:29.891798 7427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c46142b4a184014a1cebcc6fb287e9d" candidate_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" is_pre_election: true
I20260928 08:56:29.891952 7427 raft_consensus.cc:2454] T 5c46142b4a184014a1cebcc6fb287e9d P ffb3b609b7154fa4937dd34afe6adf7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6b8d92ca2404ddeb0a1a52edbae0575 in term 0.
I20260928 08:56:29.891963 7293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c46142b4a184014a1cebcc6fb287e9d" candidate_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c63387905b4bef8a4057433fedefca" is_pre_election: true
I20260928 08:56:29.892115 7293 raft_consensus.cc:2454] T 5c46142b4a184014a1cebcc6fb287e9d P e2c63387905b4bef8a4057433fedefca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6b8d92ca2404ddeb0a1a52edbae0575 in term 0.
I20260928 08:56:29.892149 7497 leader_election.cc:302] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6b8d92ca2404ddeb0a1a52edbae0575, ffb3b609b7154fa4937dd34afe6adf7b; no voters:
I20260928 08:56:29.892360 7626 raft_consensus.cc:2781] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:56:29.892400 7626 raft_consensus.cc:482] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:56:29.892416 7626 raft_consensus.cc:3037] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:56:29.893124 7626 raft_consensus.cc:504] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 44641 } } peers { permanent_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 36981 } } peers { permanent_uuid: "e2c63387905b4bef8a4057433fedefca" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46519 } }
I20260928 08:56:29.893237 7626 leader_election.cc:288] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575 [CANDIDATE]: Term 1 election: Requested vote from peers ffb3b609b7154fa4937dd34afe6adf7b (127.23.167.194:44641), e2c63387905b4bef8a4057433fedefca (127.23.167.193:46519)
I20260928 08:56:29.893400 7427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c46142b4a184014a1cebcc6fb287e9d" candidate_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffb3b609b7154fa4937dd34afe6adf7b"
I20260928 08:56:29.893419 7293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c46142b4a184014a1cebcc6fb287e9d" candidate_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c63387905b4bef8a4057433fedefca"
I20260928 08:56:29.893479 7427 raft_consensus.cc:3037] T 5c46142b4a184014a1cebcc6fb287e9d P ffb3b609b7154fa4937dd34afe6adf7b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:56:29.893522 7293 raft_consensus.cc:3037] T 5c46142b4a184014a1cebcc6fb287e9d P e2c63387905b4bef8a4057433fedefca [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:56:29.894134 7427 raft_consensus.cc:2454] T 5c46142b4a184014a1cebcc6fb287e9d P ffb3b609b7154fa4937dd34afe6adf7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6b8d92ca2404ddeb0a1a52edbae0575 in term 1.
I20260928 08:56:29.894233 7293 raft_consensus.cc:2454] T 5c46142b4a184014a1cebcc6fb287e9d P e2c63387905b4bef8a4057433fedefca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6b8d92ca2404ddeb0a1a52edbae0575 in term 1.
I20260928 08:56:29.894302 7497 leader_election.cc:302] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6b8d92ca2404ddeb0a1a52edbae0575, ffb3b609b7154fa4937dd34afe6adf7b; no voters:
I20260928 08:56:29.894388 7626 raft_consensus.cc:2781] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:56:29.894548 7626 raft_consensus.cc:686] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575 [term 1 LEADER]: Becoming Leader. State: Replica: a6b8d92ca2404ddeb0a1a52edbae0575, State: Running, Role: LEADER
I20260928 08:56:29.894636 7626 consensus_queue.cc:237] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 44641 } } peers { permanent_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 36981 } } peers { permanent_uuid: "e2c63387905b4bef8a4057433fedefca" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46519 } }
I20260928 08:56:29.895428 7147 catalog_manager.cc:5495] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6b8d92ca2404ddeb0a1a52edbae0575 (127.23.167.195). New cstate: current_term: 1 leader_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 44641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 36981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2c63387905b4bef8a4057433fedefca" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46519 } health_report { overall_health: UNKNOWN } } }
I20260928 08:56:29.911423 7617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:29.909292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49772 (local address 127.23.167.195:36981)
0928 08:56:29.909341 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:29.909343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:29.909350 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:56:29.909417 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:29.909418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:29.909434 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:29.909492 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:56:29.909590 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.910061 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.910778 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:29.910951 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:29.910993 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:29.911091 (+ 98us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:56:29.911193 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:56:29.911250 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:29.911326 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 08:56:29.911341 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 08:56:29.913269 7293 raft_consensus.cc:1261] T 5c46142b4a184014a1cebcc6fb287e9d P e2c63387905b4bef8a4057433fedefca [term 1 FOLLOWER]: Refusing update from remote peer a6b8d92ca2404ddeb0a1a52edbae0575: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:56:29.913285 7427 raft_consensus.cc:1261] T 5c46142b4a184014a1cebcc6fb287e9d P ffb3b609b7154fa4937dd34afe6adf7b [term 1 FOLLOWER]: Refusing update from remote peer a6b8d92ca2404ddeb0a1a52edbae0575: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:56:29.913615 7626 consensus_queue.cc:1035] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 44641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:56:29.913760 7631 consensus_queue.cc:1035] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c63387905b4bef8a4057433fedefca" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:56:29.916064 7634 mvcc.cc:205] Tried to move back new op lower bound from 7334239395481845760 to 7334239395409141760. Current Snapshot: MvccSnapshot[applied={T|T < 7334239395481845760}]
I20260928 08:56:29.916888 7635 mvcc.cc:205] Tried to move back new op lower bound from 7334239395481845760 to 7334239395409141760. Current Snapshot: MvccSnapshot[applied={T|T < 7334239395481845760 or (T in {7334239395481845760})}]
I20260928 08:56:29.918323 7147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:56:29.918417 7147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:56:29.919168 7147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5: Sending DeleteTablet for 3 replicas of tablet 5c46142b4a184014a1cebcc6fb287e9d
I20260928 08:56:29.919540 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 7210
I20260928 08:56:29.919523 7541 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c46142b4a184014a1cebcc6fb287e9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:56:29 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:49758
I20260928 08:56:29.919551 7273 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c46142b4a184014a1cebcc6fb287e9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:56:29 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:58016
I20260928 08:56:29.919557 7407 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c46142b4a184014a1cebcc6fb287e9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:56:29 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:45408
I20260928 08:56:29.920378 7645 tablet_replica.cc:330] T 5c46142b4a184014a1cebcc6fb287e9d P ffb3b609b7154fa4937dd34afe6adf7b: stopping tablet replica
I20260928 08:56:29.920493 7645 raft_consensus.cc:2229] T 5c46142b4a184014a1cebcc6fb287e9d P ffb3b609b7154fa4937dd34afe6adf7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:56:29.920603 7645 raft_consensus.cc:2258] T 5c46142b4a184014a1cebcc6fb287e9d P ffb3b609b7154fa4937dd34afe6adf7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:56:29.921041 7645 ts_tablet_manager.cc:1905] T 5c46142b4a184014a1cebcc6fb287e9d P ffb3b609b7154fa4937dd34afe6adf7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:56:29.922900 7645 ts_tablet_manager.cc:1918] T 5c46142b4a184014a1cebcc6fb287e9d P ffb3b609b7154fa4937dd34afe6adf7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:56:29.922964 7645 log.cc:1196] T 5c46142b4a184014a1cebcc6fb287e9d P ffb3b609b7154fa4937dd34afe6adf7b: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/wal/wals/5c46142b4a184014a1cebcc6fb287e9d
I20260928 08:56:29.923170 7645 ts_tablet_manager.cc:1939] T 5c46142b4a184014a1cebcc6fb287e9d P ffb3b609b7154fa4937dd34afe6adf7b: Deleting consensus metadata
I20260928 08:56:29.923586 7135 catalog_manager.cc:4841] TS ffb3b609b7154fa4937dd34afe6adf7b (127.23.167.194:44641): tablet 5c46142b4a184014a1cebcc6fb287e9d (table test-table [id=608b562228f64103a1401c3ba4867f40]) successfully deleted
W20260928 08:56:29.925487 7135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.167.193:46519 (error 108)
I20260928 08:56:29.925689 7644 tablet_replica.cc:330] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575: stopping tablet replica
I20260928 08:56:29.925848 7644 raft_consensus.cc:2229] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:56:29.925972 7644 raft_consensus.cc:2258] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 08:56:29.926069 7616 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:56:29.925927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:46519 (local address 127.0.0.1:58026)
0928 08:56:29.925980 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:29.926018 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.193:46519: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 08:56:29.926172 7135 catalog_manager.cc:4584] TS e2c63387905b4bef8a4057433fedefca (127.23.167.193:46519): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5c46142b4a184014a1cebcc6fb287e9d: Network error: Client connection negotiation failed: client connection to 127.23.167.193:46519: connect: Connection reset by peer (error 104)
I20260928 08:56:29.926396 7644 ts_tablet_manager.cc:1905] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:56:29.926530 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 7344
I20260928 08:56:29.928126 7644 ts_tablet_manager.cc:1918] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:56:29.928200 7644 log.cc:1196] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/wal/wals/5c46142b4a184014a1cebcc6fb287e9d
I20260928 08:56:29.928447 7644 ts_tablet_manager.cc:1939] T 5c46142b4a184014a1cebcc6fb287e9d P a6b8d92ca2404ddeb0a1a52edbae0575: Deleting consensus metadata
I20260928 08:56:29.928839 7134 catalog_manager.cc:4841] TS a6b8d92ca2404ddeb0a1a52edbae0575 (127.23.167.195:36981): tablet 5c46142b4a184014a1cebcc6fb287e9d (table test-table [id=608b562228f64103a1401c3ba4867f40]) successfully deleted
I20260928 08:56:29.932257 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 7478
I20260928 08:56:29.936944 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 7116
I20260928 08:56:49.941742 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:36677
--webserver_interface=127.23.167.254
--webserver_port=37549
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:36677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:56:50.020948 7647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:56:50.021167 7647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:56:50.021188 7647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:56:50.022363 7647 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 08:56:50.022408 7647 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 08:56:50.022419 7647 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 08:56:50.022432 7647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:56:50.022456 7647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:56:50.022480 7647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:56:50.022501 7647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:56:50.022516 7647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:56:50.023989 7647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:36677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:36677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=37549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:56:50.024226 7647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:56:50.024425 7647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:56:50.032256 7654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:50.032269 7653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:50.032450 7647 server_base.cc:1034] running on GCE node
W20260928 08:56:50.032259 7656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:50.033025 7647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:56:50.033365 7647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:56:50.034511 7647 hybrid_clock.cc:648] HybridClock initialized: now 1790585810034499 us; error 32 us; skew 500 ppm
Sep 28 08:56:50 dist-test-slave-pbjp krb5kdc[7103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585810, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:50.041025 7647 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:56:50.041484 7647 webserver.cc:460] Webserver started at http://127.23.167.254:37549/ using document root <none> and password file <none>
I20260928 08:56:50.041683 7647 fs_manager.cc:359] Metadata directory not provided
I20260928 08:56:50.041733 7647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:56:50.043018 7647 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:56:50.043619 7663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:50.043769 7647 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:56:50.043851 7647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "c6086acd2a1844d0822305982e68b0f5"
format_stamp: "Formatted at 2026-09-28 08:56:29 on dist-test-slave-pbjp"
server_key: "7b864a89a1e9e7eb8a05eeeac8d2daac"
server_key_iv: "f3e5c995f9e8e3af6be6421857091dbe"
server_key_version: "encryptionkey@0"
I20260928 08:56:50.044236 7647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:56:50.056843 7647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:56:50.058683 7647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:56:50.058818 7647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:56:50.062572 7647 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:36677
I20260928 08:56:50.062611 7715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:36677 every 8 connection(s)
I20260928 08:56:50.062963 7647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:56:50.064401 7716 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:56:50.065085 7716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5: Bootstrap starting.
I20260928 08:56:50.066351 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 7647
I20260928 08:56:50.066610 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:46519
--local_ip_for_outbound_sockets=127.23.167.193
--tserver_master_addrs=127.23.167.254:36677
--webserver_port=41123
--webserver_interface=127.23.167.193
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:56:50.073534 7716 log.cc:824] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5: Log is configured to *not* fsync() on all Append() calls
I20260928 08:56:50.075376 7716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:56:50.075650 7716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5: Bootstrap complete.
I20260928 08:56:50.077832 7716 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6086acd2a1844d0822305982e68b0f5" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 36677 } }
I20260928 08:56:50.078073 7716 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6086acd2a1844d0822305982e68b0f5, State: Initialized, Role: FOLLOWER
I20260928 08:56:50.078227 7716 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6086acd2a1844d0822305982e68b0f5" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 36677 } }
I20260928 08:56:50.078285 7716 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:56:50.078331 7716 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:56:50.078378 7716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [term 1 FOLLOWER]: Advancing to term 2
I20260928 08:56:50.079356 7716 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6086acd2a1844d0822305982e68b0f5" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 36677 } }
I20260928 08:56:50.079466 7716 leader_election.cc:302] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6086acd2a1844d0822305982e68b0f5; no voters:
I20260928 08:56:50.079627 7716 leader_election.cc:288] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 08:56:50.079680 7720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 08:56:50.079834 7720 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [term 2 LEADER]: Becoming Leader. State: Replica: c6086acd2a1844d0822305982e68b0f5, State: Running, Role: LEADER
I20260928 08:56:50.079907 7716 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:56:50.079968 7720 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6086acd2a1844d0822305982e68b0f5" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 36677 } }
I20260928 08:56:50.080277 7721 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c6086acd2a1844d0822305982e68b0f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6086acd2a1844d0822305982e68b0f5" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 36677 } } }
I20260928 08:56:50.080312 7720 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6086acd2a1844d0822305982e68b0f5. Latest consensus state: current_term: 2 leader_uuid: "c6086acd2a1844d0822305982e68b0f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6086acd2a1844d0822305982e68b0f5" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 36677 } } }
I20260928 08:56:50.080361 7721 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [sys.catalog]: This master's current role is: LEADER
I20260928 08:56:50.080403 7720 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [sys.catalog]: This master's current role is: LEADER
I20260928 08:56:50.080653 7727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:56:50.081547 7727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:56:50.081693 7727 catalog_manager.cc:1259] Loaded cluster ID: 6103a9ef8f24456697e39fc3776df251
I20260928 08:56:50.081877 7727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:56:50.083050 7727 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:56:50.083225 7727 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5: Loaded TSK: 0
I20260928 08:56:50.092916 7727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5: Generated new TSK 1
I20260928 08:56:50.093019 7727 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 08:56:50.147996 7718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:56:50.148203 7718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:56:50.148234 7718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:56:50.149500 7718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:56:50.149600 7718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:56:50.149662 7718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:56:50.151232 7718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:46519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=41123
--tserver_master_addrs=127.23.167.254:36677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:56:50.151463 7718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:56:50.151681 7718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:56:50.157651 7746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:50.157644 7743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:50.157651 7744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:50.158077 7718 server_base.cc:1034] running on GCE node
I20260928 08:56:50.158267 7718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:56:50.158510 7718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:56:50.159670 7718 hybrid_clock.cc:648] HybridClock initialized: now 1790585810159654 us; error 31 us; skew 500 ppm
Sep 28 08:56:50 dist-test-slave-pbjp krb5kdc[7103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585810, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:50.162278 7718 init.cc:377] Logged in from keytab as kudu/127.23.167.193@KRBTEST.COM (short username kudu)
I20260928 08:56:50.162745 7718 webserver.cc:460] Webserver started at http://127.23.167.193:41123/ using document root <none> and password file <none>
I20260928 08:56:50.162966 7718 fs_manager.cc:359] Metadata directory not provided
I20260928 08:56:50.163028 7718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:56:50.164352 7718 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:56:50.164989 7753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:50.165146 7718 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:56:50.165232 7718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "e2c63387905b4bef8a4057433fedefca"
format_stamp: "Formatted at 2026-09-28 08:56:29 on dist-test-slave-pbjp"
server_key: "80cf625c1f62990d4b2bedfba10fc64f"
server_key_iv: "1ad0219a8e892db7fc78781f397d7fb3"
server_key_version: "encryptionkey@0"
I20260928 08:56:50.165591 7718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:56:50.188331 7718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:56:50.190457 7718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:56:50.190586 7718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:56:50.190815 7718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:56:50.191283 7760 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 08:56:50.192775 7718 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 08:56:50.192826 7718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 08:56:50.192852 7718 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 08:56:50.193514 7718 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 08:56:50.193554 7718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 08:56:50.193617 7760 tablet_bootstrap.cc:492] T 5c46142b4a184014a1cebcc6fb287e9d P e2c63387905b4bef8a4057433fedefca: Bootstrap starting.
I20260928 08:56:50.200878 7718 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:46519
I20260928 08:56:50.201116 7867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:46519 every 8 connection(s)
I20260928 08:56:50.201450 7718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
I20260928 08:56:50.201802 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 7718
I20260928 08:56:50.202071 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:44641
--local_ip_for_outbound_sockets=127.23.167.194
--tserver_master_addrs=127.23.167.254:36677
--webserver_port=41501
--webserver_interface=127.23.167.194
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:56:50.205219 7760 log.cc:824] T 5c46142b4a184014a1cebcc6fb287e9d P e2c63387905b4bef8a4057433fedefca: Log is configured to *not* fsync() on all Append() calls
I20260928 08:56:50.207806 7760 tablet_bootstrap.cc:492] T 5c46142b4a184014a1cebcc6fb287e9d P e2c63387905b4bef8a4057433fedefca: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:56:50.208087 7760 tablet_bootstrap.cc:492] T 5c46142b4a184014a1cebcc6fb287e9d P e2c63387905b4bef8a4057433fedefca: Bootstrap complete.
I20260928 08:56:50.208829 7760 ts_tablet_manager.cc:1397] T 5c46142b4a184014a1cebcc6fb287e9d P e2c63387905b4bef8a4057433fedefca: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20260928 08:56:50.210179 7760 raft_consensus.cc:348] T 5c46142b4a184014a1cebcc6fb287e9d P e2c63387905b4bef8a4057433fedefca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 44641 } } peers { permanent_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 36981 } } peers { permanent_uuid: "e2c63387905b4bef8a4057433fedefca" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46519 } }
I20260928 08:56:50.210490 7760 raft_consensus.cc:729] T 5c46142b4a184014a1cebcc6fb287e9d P e2c63387905b4bef8a4057433fedefca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c63387905b4bef8a4057433fedefca, State: Initialized, Role: FOLLOWER
I20260928 08:56:50.210646 7760 consensus_queue.cc:260] T 5c46142b4a184014a1cebcc6fb287e9d P e2c63387905b4bef8a4057433fedefca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 44641 } } peers { permanent_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 36981 } } peers { permanent_uuid: "e2c63387905b4bef8a4057433fedefca" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46519 } }
I20260928 08:56:50.210918 7760 ts_tablet_manager.cc:1428] T 5c46142b4a184014a1cebcc6fb287e9d P e2c63387905b4bef8a4057433fedefca: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
Sep 28 08:56:50 dist-test-slave-pbjp krb5kdc[7103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585810, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:56:50.217000 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:50.202803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:48521 (local address 127.23.167.254:36677)
0928 08:56:50.205246 (+ 2443us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:50.205257 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:50.205331 (+ 74us) server_negotiation.cc:407] Connection header received
0928 08:56:50.205656 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:50.205666 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:50.205859 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:50.207151 (+ 1292us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:50.207995 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.209262 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.210921 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.211432 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.211539 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:50.213869 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:50.213873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:50.213873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:50.213883 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:50.215351 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:50.215824 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:50.215826 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:50.215826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:50.215907 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:50.216161 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:50.216161 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:50.216162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:50.216363 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:50.216420 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:50.216756 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 08:56:50.216856 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2347,"thread_start_us":70,"threads_started":1}
I20260928 08:56:50.217095 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:50.202533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.193:48521)
0928 08:56:50.202701 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:50.202744 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:50.203395 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:50.207216 (+ 3821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:50.207221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:50.207580 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:56:50.207850 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:50.207857 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.209383 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.209385 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:50.210825 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:50.210828 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.211523 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.211524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:50.211645 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:50.212145 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:56:50.212154 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:56:50.213767 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:56:50.215440 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:50.215448 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:50.215450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:50.215746 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:50.215963 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:50.215964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:50.215964 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:50.216102 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:50.216431 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:56:50.216438 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:56:50.216602 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 08:56:50.216714 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 08:56:50.216872 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
I20260928 08:56:50.217911 7868 heartbeater.cc:347] Connected to a master server at 127.23.167.254:36677
I20260928 08:56:50.218003 7868 heartbeater.cc:464] Registering TS with master...
I20260928 08:56:50.218169 7868 heartbeater.cc:511] Master 127.23.167.254:36677 requested a full tablet report, sending...
I20260928 08:56:50.218809 7680 ts_manager.cc:194] Registered new tserver with Master: e2c63387905b4bef8a4057433fedefca (127.23.167.193:46519)
I20260928 08:56:50.219796 7680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.193@KRBTEST.COM'} at 127.23.167.193:48521
I20260928 08:56:50.220839 7868 heartbeater.cc:503] Master 127.23.167.254:36677 was elected leader, sending a full tablet report...
Sep 28 08:56:50 dist-test-slave-pbjp krb5kdc[7103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585810, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for kudu/127.23.167.193@KRBTEST.COM
I20260928 08:56:50.227213 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:50.219370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:46519 (local address 127.0.0.1:34726)
0928 08:56:50.219553 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:50.219605 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:50.219666 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:50.220351 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:50.220358 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:50.220604 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:56:50.220651 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:50.220653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.221679 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.221681 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:50.222253 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:50.222255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.222694 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.222696 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:50.222762 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:50.223200 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:56:50.223204 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:56:50.224590 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:56:50.226181 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:50.226188 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:50.226190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:50.226373 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:50.226554 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:50.226555 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:50.226556 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:50.226618 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:50.226860 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:56:50.226867 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:56:50.226918 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 08:56:50.226992 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 08:56:50.227111 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":112,"threads_started":1}
I20260928 08:56:50.227236 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:50.220050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34726 (local address 127.23.167.193:46519)
0928 08:56:50.220220 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:50.220224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:50.220233 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:56:50.220263 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:50.220265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:50.220278 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:50.220345 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:50.220725 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.221610 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.222328 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.222628 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.222685 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:50.224668 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:50.224675 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:50.224675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:50.224682 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:50.226107 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:50.226445 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:50.226448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:50.226448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:50.226500 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:50.226667 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:50.226668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:50.226668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:50.226801 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:50.226872 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:50.227033 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 08:56:50.227128 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":99,"threads_started":1}
I20260928 08:56:50.227600 7802 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c46142b4a184014a1cebcc6fb287e9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:56:29 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:34726
I20260928 08:56:50.228075 7877 tablet_replica.cc:330] stopping tablet replica
I20260928 08:56:50.228180 7877 raft_consensus.cc:2229] T 5c46142b4a184014a1cebcc6fb287e9d P e2c63387905b4bef8a4057433fedefca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:56:50.228245 7877 raft_consensus.cc:2258] T 5c46142b4a184014a1cebcc6fb287e9d P e2c63387905b4bef8a4057433fedefca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:56:50.228549 7877 ts_tablet_manager.cc:1905] T 5c46142b4a184014a1cebcc6fb287e9d P e2c63387905b4bef8a4057433fedefca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:56:50.231180 7877 ts_tablet_manager.cc:1918] T 5c46142b4a184014a1cebcc6fb287e9d P e2c63387905b4bef8a4057433fedefca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:56:50.231256 7877 log.cc:1196] T 5c46142b4a184014a1cebcc6fb287e9d P e2c63387905b4bef8a4057433fedefca: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/wal/wals/5c46142b4a184014a1cebcc6fb287e9d
I20260928 08:56:50.231474 7877 ts_tablet_manager.cc:1939] T 5c46142b4a184014a1cebcc6fb287e9d P e2c63387905b4bef8a4057433fedefca: Deleting consensus metadata
I20260928 08:56:50.231946 7667 catalog_manager.cc:4841] TS e2c63387905b4bef8a4057433fedefca (127.23.167.193:46519): tablet 5c46142b4a184014a1cebcc6fb287e9d (table test-table [id=608b562228f64103a1401c3ba4867f40]) successfully deleted
W20260928 08:56:50.289461 7872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:56:50.289650 7872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:56:50.289671 7872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:56:50.291065 7872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:56:50.291113 7872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:56:50.291150 7872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:56:50.292587 7872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:44641
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=41501
--tserver_master_addrs=127.23.167.254:36677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:56:50.292804 7872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:56:50.293018 7872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:56:50.297667 7885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:50.297772 7882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:50.297709 7883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:50.297852 7872 server_base.cc:1034] running on GCE node
I20260928 08:56:50.298136 7872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:56:50.298352 7872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:56:50.299485 7872 hybrid_clock.cc:648] HybridClock initialized: now 1790585810299469 us; error 27 us; skew 500 ppm
Sep 28 08:56:50 dist-test-slave-pbjp krb5kdc[7103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585810, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:50.301800 7872 init.cc:377] Logged in from keytab as kudu/127.23.167.194@KRBTEST.COM (short username kudu)
I20260928 08:56:50.302208 7872 webserver.cc:460] Webserver started at http://127.23.167.194:41501/ using document root <none> and password file <none>
I20260928 08:56:50.302380 7872 fs_manager.cc:359] Metadata directory not provided
I20260928 08:56:50.302429 7872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:56:50.303646 7872 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:56:50.304189 7892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:50.304335 7872 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:50.304414 7872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "ffb3b609b7154fa4937dd34afe6adf7b"
format_stamp: "Formatted at 2026-09-28 08:56:29 on dist-test-slave-pbjp"
server_key: "1ee48a1a821ea765f9b6ceddbe636281"
server_key_iv: "86562fe4997eb5285122c99efe4b06f9"
server_key_version: "encryptionkey@0"
I20260928 08:56:50.304737 7872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:56:50.321246 7872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:56:50.323290 7872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:56:50.323421 7872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:56:50.323626 7872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:56:50.323907 7872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:56:50.323939 7872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:50.323977 7872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:56:50.324002 7872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:50.329488 7872 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:44641
I20260928 08:56:50.329541 8005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:44641 every 8 connection(s)
I20260928 08:56:50.329846 7872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
I20260928 08:56:50.332239 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 7872
I20260928 08:56:50.332504 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:36981
--local_ip_for_outbound_sockets=127.23.167.195
--tserver_master_addrs=127.23.167.254:36677
--webserver_port=45705
--webserver_interface=127.23.167.195
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:56:50 dist-test-slave-pbjp krb5kdc[7103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585810, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:56:50.339159 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:50.330500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:58275 (local address 127.23.167.254:36677)
0928 08:56:50.330551 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:50.330553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:50.330721 (+ 168us) server_negotiation.cc:407] Connection header received
0928 08:56:50.331206 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:50.331207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:50.331220 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:50.331269 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:50.332148 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.332852 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.334045 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.334252 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.334310 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:50.336471 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:50.336473 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:50.336473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:50.336480 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:50.337745 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:50.338227 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:50.338229 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:50.338229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:50.338288 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:50.338556 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:50.338556 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:50.338556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:50.338662 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:50.338709 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:50.339019 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 08:56:50.339094 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 08:56:50.339435 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:50.330473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.194:58275)
0928 08:56:50.330621 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:50.330655 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:50.331125 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:50.331324 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:50.331327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:50.331863 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:56:50.332053 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:50.332058 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.332937 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.332938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:50.333953 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:50.333956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.334322 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.334323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:50.334452 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:50.334934 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:56:50.334943 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:56:50.336370 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:56:50.337821 (+ 1451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:50.337829 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:50.337831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:50.338133 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:50.338355 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:50.338356 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:50.338356 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:50.338495 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:50.338718 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:56:50.338724 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:56:50.338892 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 08:56:50.339013 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 08:56:50.339184 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260928 08:56:50.340037 8006 heartbeater.cc:347] Connected to a master server at 127.23.167.254:36677
I20260928 08:56:50.340142 8006 heartbeater.cc:464] Registering TS with master...
I20260928 08:56:50.340343 8006 heartbeater.cc:511] Master 127.23.167.254:36677 requested a full tablet report, sending...
I20260928 08:56:50.340783 7680 ts_manager.cc:194] Registered new tserver with Master: ffb3b609b7154fa4937dd34afe6adf7b (127.23.167.194:44641)
I20260928 08:56:50.341300 7680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.194@KRBTEST.COM'} at 127.23.167.194:58275
W20260928 08:56:50.412505 8009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:56:50.412703 8009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:56:50.412724 8009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:56:50.413810 8009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:56:50.413841 8009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:56:50.413864 8009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:56:50.415231 8009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:36981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=45705
--tserver_master_addrs=127.23.167.254:36677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:56:50.415405 8009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:56:50.415576 8009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:56:50.420231 8017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:56:50.420241 8014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:50.420356 8009 server_base.cc:1034] running on GCE node
W20260928 08:56:50.420248 8015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:56:50.420644 8009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:56:50.420845 8009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:56:50.421968 8009 hybrid_clock.cc:648] HybridClock initialized: now 1790585810421944 us; error 39 us; skew 500 ppm
Sep 28 08:56:50 dist-test-slave-pbjp krb5kdc[7103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585810, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:56:50.424178 8009 init.cc:377] Logged in from keytab as kudu/127.23.167.195@KRBTEST.COM (short username kudu)
I20260928 08:56:50.424556 8009 webserver.cc:460] Webserver started at http://127.23.167.195:45705/ using document root <none> and password file <none>
I20260928 08:56:50.424762 8009 fs_manager.cc:359] Metadata directory not provided
I20260928 08:56:50.424821 8009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:56:50.426188 8009 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:56:50.426753 8024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:50.426919 8009 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:56:50.427006 8009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "a6b8d92ca2404ddeb0a1a52edbae0575"
format_stamp: "Formatted at 2026-09-28 08:56:29 on dist-test-slave-pbjp"
server_key: "c81c39ea847e05682a2f036b23a46744"
server_key_iv: "a9ffed92561b9e94d80b335377750023"
server_key_version: "encryptionkey@0"
I20260928 08:56:50.427320 8009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:56:50.441576 8009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:56:50.443262 8009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:56:50.443390 8009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:56:50.443611 8009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:56:50.443886 8009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:56:50.443917 8009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:50.443946 8009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:56:50.443970 8009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:56:50.449370 8009 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:36981
I20260928 08:56:50.449424 8137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:36981 every 8 connection(s)
I20260928 08:56:50.449738 8009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
Sep 28 08:56:50 dist-test-slave-pbjp krb5kdc[7103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585810, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:56:50.457365 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 8009
I20260928 08:56:50.458989 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:50.450454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:52767 (local address 127.23.167.254:36677)
0928 08:56:50.450555 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:50.450557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:50.450797 (+ 240us) server_negotiation.cc:407] Connection header received
0928 08:56:50.451458 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:50.451459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:50.451472 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:50.451526 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:50.452782 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.453447 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.454389 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.454575 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.454629 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:50.456488 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:50.456489 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:50.456489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:50.456496 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:50.457731 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:50.458114 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:50.458116 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:50.458116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:50.458175 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:50.458385 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:50.458386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:50.458386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:50.458498 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:50.458552 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:50.458822 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 08:56:50.458903 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 08:56:50.459250 8140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:50.450437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.195:52767)
0928 08:56:50.450618 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:50.450693 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:50.451341 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:50.451609 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:50.451614 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:50.452438 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:56:50.452695 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:50.452702 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.453519 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.453520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:50.454319 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:50.454321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.454634 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.454634 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:50.454723 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:50.455127 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:56:50.455134 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:56:50.456412 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:56:50.457805 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:50.457812 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:50.457814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:50.458024 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:50.458232 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:56:50.458233 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:56:50.458233 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:56:50.458332 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:56:50.458559 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:56:50.458573 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:56:50.458705 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 08:56:50.458792 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 08:56:50.458939 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
I20260928 08:56:50.459695 8138 heartbeater.cc:347] Connected to a master server at 127.23.167.254:36677
I20260928 08:56:50.459779 8138 heartbeater.cc:464] Registering TS with master...
I20260928 08:56:50.459942 8138 heartbeater.cc:511] Master 127.23.167.254:36677 requested a full tablet report, sending...
I20260928 08:56:50.460299 7680 ts_manager.cc:194] Registered new tserver with Master: a6b8d92ca2404ddeb0a1a52edbae0575 (127.23.167.195:36981)
I20260928 08:56:50.460712 7680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.195@KRBTEST.COM'} at 127.23.167.195:52767
I20260928 08:56:50.463289 8142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:50.457749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37200 (local address 127.23.167.254:36677)
0928 08:56:50.457859 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:50.457860 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:50.457931 (+ 71us) server_negotiation.cc:407] Connection header received
0928 08:56:50.457989 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:50.457990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:50.458000 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:50.458061 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:50.458595 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.459065 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.459998 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.460320 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.460378 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:50.461459 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:50.461460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:50.461460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:50.461466 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:50.462488 (+ 1022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:50.462756 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:50.462757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:50.462757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:50.462800 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:50.462934 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:50.462934 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:50.462934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:50.463015 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:50.463142 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:50.463168 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 08:56:50.463225 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":57,"threads_started":1}
I20260928 08:56:50.463930 24223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:56:50.466270 8142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:50.464217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37216 (local address 127.23.167.254:36677)
0928 08:56:50.464270 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:50.464271 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:50.464356 (+ 85us) server_negotiation.cc:407] Connection header received
0928 08:56:50.464403 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:50.464404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:50.464430 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:50.464480 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:56:50.464541 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.464969 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.465695 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.465881 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.465938 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:50.466029 (+ 91us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:56:50.466066 (+ 37us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 08:56:50.466217 (+ 151us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37216: token expired
Metrics: {"server-negotiator.queue_time_us":32}
W20260928 08:56:50.466326 8142 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37216: token expired
W20260928 08:56:50.466370 7613 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.23.167.254:36677)
I20260928 08:56:50.471426 8142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:50.466580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37228 (local address 127.23.167.254:36677)
0928 08:56:50.466618 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:50.466620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:50.466628 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:56:50.466679 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:50.466680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:50.466688 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:50.466733 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:56:50.467136 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.467607 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.468241 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.468463 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.468516 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:50.469523 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:56:50.469524 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:56:50.469524 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:56:50.469531 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:56:50.470523 (+ 992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:50.470799 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:50.470800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:50.470800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:50.470849 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:56:50.471005 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:56:50.471005 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:56:50.471006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:56:50.471106 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:56:50.471188 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:50.471278 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 08:56:50.471346 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 08:56:50.472378 7680 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:56:50.474032 7680 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:56:50.476074 7802 tablet_service.cc:1467] Processing CreateTablet for tablet b6f55a780d074d6eb6dad0a13e0fc27a (DEFAULT_TABLE table=test-table [id=2cf91f7a991147818143a6c9881e8401]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:56:50.476251 7802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6f55a780d074d6eb6dad0a13e0fc27a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:56:50.477682 7760 tablet_bootstrap.cc:492] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca: Bootstrap starting.
I20260928 08:56:50.478004 7760 tablet_bootstrap.cc:654] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca: Neither blocks nor log segments found. Creating new log.
I20260928 08:56:50.478529 7760 tablet_bootstrap.cc:492] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca: No bootstrap required, opened a new log
I20260928 08:56:50.478581 7760 ts_tablet_manager.cc:1397] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 08:56:50.478919 7760 raft_consensus.cc:348] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 36981 } } peers { permanent_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 44641 } } peers { permanent_uuid: "e2c63387905b4bef8a4057433fedefca" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46519 } }
I20260928 08:56:50.478999 7760 raft_consensus.cc:374] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:56:50.479043 7760 raft_consensus.cc:729] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c63387905b4bef8a4057433fedefca, State: Initialized, Role: FOLLOWER
I20260928 08:56:50.479084 7760 consensus_queue.cc:260] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 36981 } } peers { permanent_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 44641 } } peers { permanent_uuid: "e2c63387905b4bef8a4057433fedefca" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46519 } }
I20260928 08:56:50.479166 7760 ts_tablet_manager.cc:1428] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 08:56:50.479809 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:50.475657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:36981 (local address 127.0.0.1:54598)
0928 08:56:50.475705 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:50.475715 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:50.475754 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:50.476147 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:50.476148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:50.476150 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:56:50.476193 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:50.476195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.477348 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.477349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:50.478388 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:50.478392 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.479519 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.479520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:50.479599 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:50.479620 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 08:56:50.479681 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 08:56:50.479727 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 08:56:50.479876 8145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:50.475738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54598 (local address 127.23.167.195:36981)
0928 08:56:50.475997 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:50.476003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:50.476012 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:56:50.476041 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:50.476042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:50.476055 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:50.476126 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:56:50.476254 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.477251 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.478538 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.479435 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.479520 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:50.479579 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:50.479753 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 08:56:50.479804 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":105,"threads_started":1}
I20260928 08:56:50.480535 8144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:50.475701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:44641 (local address 127.0.0.1:41242)
0928 08:56:50.475843 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:50.475853 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:50.475902 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:50.477509 (+ 1607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:50.477510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:50.477512 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:56:50.477546 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:50.477547 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.478548 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.478549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:50.479506 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:50.479508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.480313 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.480314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:50.480375 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:50.480379 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:56:50.480436 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 08:56:50.480473 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
I20260928 08:56:50.480513 8072 tablet_service.cc:1467] Processing CreateTablet for tablet b6f55a780d074d6eb6dad0a13e0fc27a (DEFAULT_TABLE table=test-table [id=2cf91f7a991147818143a6c9881e8401]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:56:50.480577 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:50.477167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41242 (local address 127.23.167.194:44641)
0928 08:56:50.477349 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:50.477354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:50.477364 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:56:50.477400 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:50.477402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:50.477417 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:50.477506 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:56:50.477601 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.478475 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.479599 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.480241 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.480299 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:50.480336 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:50.480484 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 08:56:50.480515 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":82,"threads_started":1}
I20260928 08:56:50.480863 8072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6f55a780d074d6eb6dad0a13e0fc27a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:56:50.481369 7940 tablet_service.cc:1467] Processing CreateTablet for tablet b6f55a780d074d6eb6dad0a13e0fc27a (DEFAULT_TABLE table=test-table [id=2cf91f7a991147818143a6c9881e8401]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:56:50.481669 7940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6f55a780d074d6eb6dad0a13e0fc27a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:56:50.483457 8148 tablet_bootstrap.cc:492] T b6f55a780d074d6eb6dad0a13e0fc27a P a6b8d92ca2404ddeb0a1a52edbae0575: Bootstrap starting.
I20260928 08:56:50.484002 8148 tablet_bootstrap.cc:654] T b6f55a780d074d6eb6dad0a13e0fc27a P a6b8d92ca2404ddeb0a1a52edbae0575: Neither blocks nor log segments found. Creating new log.
I20260928 08:56:50.484165 8149 tablet_bootstrap.cc:492] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b: Bootstrap starting.
I20260928 08:56:50.484519 8148 log.cc:824] T b6f55a780d074d6eb6dad0a13e0fc27a P a6b8d92ca2404ddeb0a1a52edbae0575: Log is configured to *not* fsync() on all Append() calls
I20260928 08:56:50.484867 8149 tablet_bootstrap.cc:654] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b: Neither blocks nor log segments found. Creating new log.
I20260928 08:56:50.485281 8148 tablet_bootstrap.cc:492] T b6f55a780d074d6eb6dad0a13e0fc27a P a6b8d92ca2404ddeb0a1a52edbae0575: No bootstrap required, opened a new log
I20260928 08:56:50.485343 8148 ts_tablet_manager.cc:1397] T b6f55a780d074d6eb6dad0a13e0fc27a P a6b8d92ca2404ddeb0a1a52edbae0575: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:56:50.485399 8149 log.cc:824] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b: Log is configured to *not* fsync() on all Append() calls
I20260928 08:56:50.486068 8149 tablet_bootstrap.cc:492] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b: No bootstrap required, opened a new log
I20260928 08:56:50.486119 8149 ts_tablet_manager.cc:1397] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 08:56:50.486809 8148 raft_consensus.cc:348] T b6f55a780d074d6eb6dad0a13e0fc27a P a6b8d92ca2404ddeb0a1a52edbae0575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 36981 } } peers { permanent_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 44641 } } peers { permanent_uuid: "e2c63387905b4bef8a4057433fedefca" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46519 } }
I20260928 08:56:50.486953 8148 raft_consensus.cc:374] T b6f55a780d074d6eb6dad0a13e0fc27a P a6b8d92ca2404ddeb0a1a52edbae0575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:56:50.486985 8148 raft_consensus.cc:729] T b6f55a780d074d6eb6dad0a13e0fc27a P a6b8d92ca2404ddeb0a1a52edbae0575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6b8d92ca2404ddeb0a1a52edbae0575, State: Initialized, Role: FOLLOWER
I20260928 08:56:50.487099 8148 consensus_queue.cc:260] T b6f55a780d074d6eb6dad0a13e0fc27a P a6b8d92ca2404ddeb0a1a52edbae0575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 36981 } } peers { permanent_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 44641 } } peers { permanent_uuid: "e2c63387905b4bef8a4057433fedefca" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46519 } }
I20260928 08:56:50.487349 8148 ts_tablet_manager.cc:1428] T b6f55a780d074d6eb6dad0a13e0fc27a P a6b8d92ca2404ddeb0a1a52edbae0575: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 08:56:50.487372 8138 heartbeater.cc:503] Master 127.23.167.254:36677 was elected leader, sending a full tablet report...
I20260928 08:56:50.487530 8149 raft_consensus.cc:348] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 36981 } } peers { permanent_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 44641 } } peers { permanent_uuid: "e2c63387905b4bef8a4057433fedefca" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46519 } }
I20260928 08:56:50.487686 8149 raft_consensus.cc:374] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:56:50.487728 8149 raft_consensus.cc:729] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffb3b609b7154fa4937dd34afe6adf7b, State: Initialized, Role: FOLLOWER
I20260928 08:56:50.487830 8149 consensus_queue.cc:260] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 36981 } } peers { permanent_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 44641 } } peers { permanent_uuid: "e2c63387905b4bef8a4057433fedefca" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46519 } }
I20260928 08:56:50.488081 8149 ts_tablet_manager.cc:1428] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 08:56:50.488127 8006 heartbeater.cc:503] Master 127.23.167.254:36677 was elected leader, sending a full tablet report...
W20260928 08:56:50.580346 8007 tablet.cc:2369] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:56:50.700289 8139 tablet.cc:2369] T b6f55a780d074d6eb6dad0a13e0fc27a P a6b8d92ca2404ddeb0a1a52edbae0575: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:56:50.702292 7869 tablet.cc:2369] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:56:50.742756 8153 raft_consensus.cc:482] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:56:50.742894 8153 raft_consensus.cc:504] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 36981 } } peers { permanent_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 44641 } } peers { permanent_uuid: "e2c63387905b4bef8a4057433fedefca" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46519 } }
I20260928 08:56:50.743261 8153 leader_election.cc:288] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6b8d92ca2404ddeb0a1a52edbae0575 (127.23.167.195:36981), e2c63387905b4bef8a4057433fedefca (127.23.167.193:46519)
I20260928 08:56:50.746131 8145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:50.743506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:49133 (local address 127.23.167.195:36981)
0928 08:56:50.743547 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:50.743549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:50.743656 (+ 107us) server_negotiation.cc:407] Connection header received
0928 08:56:50.743692 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:50.743693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:50.743703 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:50.743754 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:56:50.743915 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.744417 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.745415 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.745801 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.745854 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:50.745861 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:50.746018 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 08:56:50.746056 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 08:56:50.746138 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:50.743452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:36981 (local address 127.23.167.194:49133)
0928 08:56:50.743532 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:50.743543 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:50.743603 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:50.743768 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:50.743769 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:50.743771 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:56:50.743850 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:50.743852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.744498 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.744499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:50.745331 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:50.745333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.745875 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.745875 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:50.745954 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:50.745958 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:56:50.746006 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 08:56:50.746044 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 08:56:50.746250 8154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:50.743464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:46519 (local address 127.23.167.194:46385)
0928 08:56:50.743643 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:50.743654 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:56:50.743714 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:56:50.743891 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:56:50.743892 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:56:50.743893 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:56:50.743966 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:50.743967 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.744573 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.744574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:50.745399 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:56:50.745402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.745983 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.745983 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:56:50.746052 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:50.746056 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:56:50.746118 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 08:56:50.746153 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":17,"thread_start_us":95,"threads_started":1}
I20260928 08:56:50.746456 8155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:50.743508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:46385 (local address 127.23.167.193:46519)
0928 08:56:50.743637 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:50.743639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:50.743711 (+ 72us) server_negotiation.cc:407] Connection header received
0928 08:56:50.743771 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:50.743771 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:50.743802 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:50.743885 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:56:50.744026 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.744494 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.745480 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.745906 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.745979 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:50.746053 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:50.746128 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 08:56:50.746168 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260928 08:56:50.746500 8092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6f55a780d074d6eb6dad0a13e0fc27a" candidate_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" is_pre_election: true
I20260928 08:56:50.746659 8092 raft_consensus.cc:2454] T b6f55a780d074d6eb6dad0a13e0fc27a P a6b8d92ca2404ddeb0a1a52edbae0575 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffb3b609b7154fa4937dd34afe6adf7b in term 0.
I20260928 08:56:50.746726 7822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6f55a780d074d6eb6dad0a13e0fc27a" candidate_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c63387905b4bef8a4057433fedefca" is_pre_election: true
I20260928 08:56:50.746855 7822 raft_consensus.cc:2454] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffb3b609b7154fa4937dd34afe6adf7b in term 0.
I20260928 08:56:50.746918 7895 leader_election.cc:302] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6b8d92ca2404ddeb0a1a52edbae0575, ffb3b609b7154fa4937dd34afe6adf7b; no voters:
I20260928 08:56:50.747098 8153 raft_consensus.cc:2781] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:56:50.747138 8153 raft_consensus.cc:482] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:56:50.747152 8153 raft_consensus.cc:3037] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:56:50.747947 8153 raft_consensus.cc:504] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 36981 } } peers { permanent_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 44641 } } peers { permanent_uuid: "e2c63387905b4bef8a4057433fedefca" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46519 } }
I20260928 08:56:50.748059 8153 leader_election.cc:288] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b [CANDIDATE]: Term 1 election: Requested vote from peers a6b8d92ca2404ddeb0a1a52edbae0575 (127.23.167.195:36981), e2c63387905b4bef8a4057433fedefca (127.23.167.193:46519)
I20260928 08:56:50.748221 7822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6f55a780d074d6eb6dad0a13e0fc27a" candidate_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c63387905b4bef8a4057433fedefca"
I20260928 08:56:50.748230 8092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6f55a780d074d6eb6dad0a13e0fc27a" candidate_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575"
I20260928 08:56:50.748291 7822 raft_consensus.cc:3037] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:56:50.748302 8092 raft_consensus.cc:3037] T b6f55a780d074d6eb6dad0a13e0fc27a P a6b8d92ca2404ddeb0a1a52edbae0575 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:56:50.748894 7822 raft_consensus.cc:2454] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffb3b609b7154fa4937dd34afe6adf7b in term 1.
I20260928 08:56:50.749008 8092 raft_consensus.cc:2454] T b6f55a780d074d6eb6dad0a13e0fc27a P a6b8d92ca2404ddeb0a1a52edbae0575 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffb3b609b7154fa4937dd34afe6adf7b in term 1.
I20260928 08:56:50.749080 7896 leader_election.cc:302] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e2c63387905b4bef8a4057433fedefca, ffb3b609b7154fa4937dd34afe6adf7b; no voters:
I20260928 08:56:50.749191 8153 raft_consensus.cc:2781] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:56:50.749339 8153 raft_consensus.cc:686] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b [term 1 LEADER]: Becoming Leader. State: Replica: ffb3b609b7154fa4937dd34afe6adf7b, State: Running, Role: LEADER
I20260928 08:56:50.749425 8153 consensus_queue.cc:237] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 36981 } } peers { permanent_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 44641 } } peers { permanent_uuid: "e2c63387905b4bef8a4057433fedefca" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46519 } }
I20260928 08:56:50.750242 7680 catalog_manager.cc:5495] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b reported cstate change: term changed from 0 to 1, leader changed from <none> to ffb3b609b7154fa4937dd34afe6adf7b (127.23.167.194). New cstate: current_term: 1 leader_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 36981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 44641 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2c63387905b4bef8a4057433fedefca" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46519 } health_report { overall_health: UNKNOWN } } }
I20260928 08:56:50.759994 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:50.758003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41252 (local address 127.23.167.194:44641)
0928 08:56:50.758053 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 08:56:50.758055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:56:50.758061 (+ 6us) server_negotiation.cc:407] Connection header received
0928 08:56:50.758091 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:56:50.758092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:56:50.758109 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:56:50.758165 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:56:50.758258 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.758715 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.759401 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:56:50.759575 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:56:50.759638 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:56:50.759754 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:56:50.759845 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:56:50.759898 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 08:56:50.759913 (+ 15us) server_negotiation.cc:299] Negotiation successful
0928 08:56:50.759928 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 08:56:50.761610 8092 raft_consensus.cc:1261] T b6f55a780d074d6eb6dad0a13e0fc27a P a6b8d92ca2404ddeb0a1a52edbae0575 [term 1 FOLLOWER]: Refusing update from remote peer ffb3b609b7154fa4937dd34afe6adf7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:56:50.761608 7822 raft_consensus.cc:1261] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [term 1 FOLLOWER]: Refusing update from remote peer ffb3b609b7154fa4937dd34afe6adf7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:56:50.761895 8153 consensus_queue.cc:1035] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c63387905b4bef8a4057433fedefca" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:56:50.762024 8156 consensus_queue.cc:1035] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 36981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:56:50.764179 8160 mvcc.cc:205] Tried to move back new op lower bound from 7334239480876957696 to 7334239480830156800. Current Snapshot: MvccSnapshot[applied={T|T < 7334239480876957696}]
I20260928 08:56:50.764883 8162 mvcc.cc:205] Tried to move back new op lower bound from 7334239480876957696 to 7334239480830156800. Current Snapshot: MvccSnapshot[applied={T|T < 7334239480876957696}]
I20260928 08:56:50.766096 7679 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:56:50.766191 7679 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:56:50.766860 7679 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5: Sending DeleteTablet for 3 replicas of tablet b6f55a780d074d6eb6dad0a13e0fc27a
I20260928 08:56:50.767136 7802 tablet_service.cc:1514] Processing DeleteTablet for tablet b6f55a780d074d6eb6dad0a13e0fc27a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:56:50 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:34726
I20260928 08:56:50.767155 7940 tablet_service.cc:1514] Processing DeleteTablet for tablet b6f55a780d074d6eb6dad0a13e0fc27a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:56:50 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:41242
I20260928 08:56:50.767349 8168 tablet_replica.cc:330] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca: stopping tablet replica
I20260928 08:56:50.767396 8168 raft_consensus.cc:2229] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:56:50.767442 8168 raft_consensus.cc:2258] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:56:50.767436 8072 tablet_service.cc:1514] Processing DeleteTablet for tablet b6f55a780d074d6eb6dad0a13e0fc27a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:56:50 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:54598
I20260928 08:56:50.767630 8170 tablet_replica.cc:330] T b6f55a780d074d6eb6dad0a13e0fc27a P a6b8d92ca2404ddeb0a1a52edbae0575: stopping tablet replica
I20260928 08:56:50.767740 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 7718
I20260928 08:56:50.767807 8168 ts_tablet_manager.cc:1905] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:56:50.767772 8169 tablet_replica.cc:330] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b: stopping tablet replica
I20260928 08:56:50.768457 8169 raft_consensus.cc:2229] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:56:50.768559 8169 raft_consensus.cc:2258] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:56:50.768973 8169 ts_tablet_manager.cc:1905] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:56:50.770982 8169 ts_tablet_manager.cc:1918] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:56:50.771055 8169 log.cc:1196] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/wal/wals/b6f55a780d074d6eb6dad0a13e0fc27a
I20260928 08:56:50.771353 8169 ts_tablet_manager.cc:1939] T b6f55a780d074d6eb6dad0a13e0fc27a P ffb3b609b7154fa4937dd34afe6adf7b: Deleting consensus metadata
I20260928 08:56:50.771770 7667 catalog_manager.cc:4841] TS ffb3b609b7154fa4937dd34afe6adf7b (127.23.167.194:44641): tablet b6f55a780d074d6eb6dad0a13e0fc27a (table test-table [id=2cf91f7a991147818143a6c9881e8401]) successfully deleted
W20260928 08:56:50.772903 7667 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.167.193:46519 (error 108)
I20260928 08:56:50.773267 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 7872
I20260928 08:56:50.773388 8144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:50.773264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:46519 (local address 127.0.0.1:34740)
0928 08:56:50.773320 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:50.773358 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.193:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 08:56:50.773814 7667 catalog_manager.cc:4584] TS e2c63387905b4bef8a4057433fedefca (127.23.167.193:46519): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b6f55a780d074d6eb6dad0a13e0fc27a: Network error: Client connection negotiation failed: client connection to 127.23.167.193:46519: connect: Connection refused (error 111)
I20260928 08:56:50.777209 8170 raft_consensus.cc:2229] T b6f55a780d074d6eb6dad0a13e0fc27a P a6b8d92ca2404ddeb0a1a52edbae0575 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:56:50.777348 8170 raft_consensus.cc:2258] T b6f55a780d074d6eb6dad0a13e0fc27a P a6b8d92ca2404ddeb0a1a52edbae0575 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:56:50.777747 8170 ts_tablet_manager.cc:1905] T b6f55a780d074d6eb6dad0a13e0fc27a P a6b8d92ca2404ddeb0a1a52edbae0575: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:56:50.778566 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 8009
I20260928 08:56:50.782817 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 7647
I20260928 08:56:50.782956 8144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:56:50.782790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:36981 (local address 127.0.0.1:54602)
0928 08:56:50.782875 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 08:56:50.782905 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.195:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 08:57:10.787431 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:36677
--webserver_interface=127.23.167.254
--webserver_port=37549
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:36677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:10.868165 8172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:10.868378 8172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:10.868399 8172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:10.869588 8172 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 08:57:10.869625 8172 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 08:57:10.869637 8172 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 08:57:10.869647 8172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:57:10.869658 8172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:10.869669 8172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:10.869683 8172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:57:10.869702 8172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:57:10.871192 8172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:36677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:36677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=37549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:10.871419 8172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:10.871644 8172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:10.876298 8178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:10.876261 8180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:10.876272 8177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:10.876475 8172 server_base.cc:1034] running on GCE node
I20260928 08:57:10.876756 8172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:10.877023 8172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:10.878187 8172 hybrid_clock.cc:648] HybridClock initialized: now 1790585830878171 us; error 34 us; skew 500 ppm
Sep 28 08:57:10 dist-test-slave-pbjp krb5kdc[7103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585830, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:10.885804 8172 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:57:10.886235 8172 webserver.cc:460] Webserver started at http://127.23.167.254:37549/ using document root <none> and password file <none>
I20260928 08:57:10.886423 8172 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:10.886463 8172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:10.887693 8172 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:57:10.888265 8187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:10.888429 8172 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:57:10.888509 8172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "c6086acd2a1844d0822305982e68b0f5"
format_stamp: "Formatted at 2026-09-28 08:56:29 on dist-test-slave-pbjp"
server_key: "7b864a89a1e9e7eb8a05eeeac8d2daac"
server_key_iv: "f3e5c995f9e8e3af6be6421857091dbe"
server_key_version: "encryptionkey@0"
I20260928 08:57:10.888832 8172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:10.907477 8172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:10.909603 8172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:10.909716 8172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:10.913504 8172 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:36677
I20260928 08:57:10.913573 8239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:36677 every 8 connection(s)
I20260928 08:57:10.913923 8172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:57:10.915463 8240 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:57:10.916126 8240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5: Bootstrap starting.
I20260928 08:57:10.922506 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 8172
I20260928 08:57:10.922869 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:46519
--local_ip_for_outbound_sockets=127.23.167.193
--tserver_master_addrs=127.23.167.254:36677
--webserver_port=41123
--webserver_interface=127.23.167.193
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:57:10.925207 8240 log.cc:824] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:10.927491 8240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:57:10.927783 8240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5: Bootstrap complete.
I20260928 08:57:10.930063 8240 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6086acd2a1844d0822305982e68b0f5" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 36677 } }
I20260928 08:57:10.930302 8240 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6086acd2a1844d0822305982e68b0f5, State: Initialized, Role: FOLLOWER
I20260928 08:57:10.930455 8240 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6086acd2a1844d0822305982e68b0f5" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 36677 } }
I20260928 08:57:10.930526 8240 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:57:10.930568 8240 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:57:10.930614 8240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [term 2 FOLLOWER]: Advancing to term 3
I20260928 08:57:10.931588 8240 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6086acd2a1844d0822305982e68b0f5" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 36677 } }
I20260928 08:57:10.931674 8240 leader_election.cc:302] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6086acd2a1844d0822305982e68b0f5; no voters:
I20260928 08:57:10.931874 8240 leader_election.cc:288] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 08:57:10.931923 8244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 08:57:10.932041 8244 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [term 3 LEADER]: Becoming Leader. State: Replica: c6086acd2a1844d0822305982e68b0f5, State: Running, Role: LEADER
I20260928 08:57:10.932137 8240 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:57:10.932124 8244 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6086acd2a1844d0822305982e68b0f5" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 36677 } }
I20260928 08:57:10.932426 8246 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c6086acd2a1844d0822305982e68b0f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6086acd2a1844d0822305982e68b0f5" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 36677 } } }
I20260928 08:57:10.932428 8244 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6086acd2a1844d0822305982e68b0f5. Latest consensus state: current_term: 3 leader_uuid: "c6086acd2a1844d0822305982e68b0f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6086acd2a1844d0822305982e68b0f5" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 36677 } } }
I20260928 08:57:10.932652 8244 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:10.932847 8246 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5 [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:10.932991 8253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:57:10.933879 8253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:57:10.934007 8253 catalog_manager.cc:1259] Loaded cluster ID: 6103a9ef8f24456697e39fc3776df251
I20260928 08:57:10.934042 8253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:57:10.935467 8253 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:57:10.935660 8253 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5: Loaded TSK: 0
I20260928 08:57:10.935706 8253 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c6086acd2a1844d0822305982e68b0f5: Loaded TSK: 1
F20260928 08:57:10.935956 8253 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd65ddd69ad google::LogMessage::Fail() at ??:0
@ 0x7fd65ddd988a google::LogMessage::SendToLog() at ??:0
@ 0x7fd65ddd64de google::LogMessage::Flush() at ??:0
@ 0x7fd65ddd7a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd660543104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fd660540d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fd665d301cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fd665d3048e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fd665d30594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fd665cf6c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fd665d00ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd65fca40d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd65fc99954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd660afd6db start_thread at ??:0
@ 0x7fd65c8d461f clone at ??:0
W20260928 08:57:11.032853 8242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:11.033056 8242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:11.033118 8242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:11.034273 8242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:11.034312 8242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:11.034337 8242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:57:11.035764 8242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:46519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=41123
--tserver_master_addrs=127.23.167.254:36677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:11.035985 8242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:11.036202 8242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:11.041296 8284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:11.041301 8281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:11.041312 8282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:11.041765 8242 server_base.cc:1034] running on GCE node
I20260928 08:57:11.042007 8242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:11.042253 8242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:11.043407 8242 hybrid_clock.cc:648] HybridClock initialized: now 1790585831043392 us; error 34 us; skew 500 ppm
Sep 28 08:57:11 dist-test-slave-pbjp krb5kdc[7103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585831, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:11.045893 8242 init.cc:377] Logged in from keytab as kudu/127.23.167.193@KRBTEST.COM (short username kudu)
I20260928 08:57:11.046343 8242 webserver.cc:460] Webserver started at http://127.23.167.193:41123/ using document root <none> and password file <none>
I20260928 08:57:11.046557 8242 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:11.046607 8242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:11.047983 8242 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:57:11.048740 8291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:11.048892 8242 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:57:11.048977 8242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "e2c63387905b4bef8a4057433fedefca"
format_stamp: "Formatted at 2026-09-28 08:56:29 on dist-test-slave-pbjp"
server_key: "80cf625c1f62990d4b2bedfba10fc64f"
server_key_iv: "1ad0219a8e892db7fc78781f397d7fb3"
server_key_version: "encryptionkey@0"
I20260928 08:57:11.049356 8242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:11.071676 8242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:11.073910 8242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:11.074059 8242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:11.074285 8242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:57:11.074723 8298 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 08:57:11.076611 8242 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 08:57:11.076659 8242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 08:57:11.076694 8242 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 08:57:11.077348 8242 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 08:57:11.077381 8242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 08:57:11.077471 8298 tablet_bootstrap.cc:492] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca: Bootstrap starting.
I20260928 08:57:11.079385 8298 log.cc:824] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:11.082298 8298 tablet_bootstrap.cc:492] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:57:11.082608 8298 tablet_bootstrap.cc:492] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca: Bootstrap complete.
I20260928 08:57:11.082762 8298 ts_tablet_manager.cc:1397] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 08:57:11.084008 8242 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:46519
I20260928 08:57:11.084110 8406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:46519 every 8 connection(s)
I20260928 08:57:11.084359 8242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
I20260928 08:57:11.084539 8298 raft_consensus.cc:348] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 36981 } } peers { permanent_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 44641 } } peers { permanent_uuid: "e2c63387905b4bef8a4057433fedefca" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46519 } }
I20260928 08:57:11.085479 8298 raft_consensus.cc:729] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c63387905b4bef8a4057433fedefca, State: Initialized, Role: FOLLOWER
I20260928 08:57:11.085615 8298 consensus_queue.cc:260] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 36981 } } peers { permanent_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 44641 } } peers { permanent_uuid: "e2c63387905b4bef8a4057433fedefca" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46519 } }
I20260928 08:57:11.085883 8298 ts_tablet_manager.cc:1428] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 08:57:11.088621 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 8242
I20260928 08:57:11.088862 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:44641
--local_ip_for_outbound_sockets=127.23.167.194
--tserver_master_addrs=127.23.167.254:36677
--webserver_port=41501
--webserver_interface=127.23.167.194
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:11.149259 8409 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:57:11.085044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.193:46491)
0928 08:57:11.085222 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:11.085257 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:11.085379 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:11.149013 (+ 63634us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
W20260928 08:57:11.149526 8407 heartbeater.cc:644] Failed to heartbeat to 127.23.167.254:36677 (0 consecutive failures): Network error: Failed to ping master at 127.23.167.254:36677: Client connection negotiation failed: client connection to 127.23.167.254:36677: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 08:57:11.149917 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:11.149796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.193:51029)
0928 08:57:11.149849 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:11.149893 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 08:57:11.150261 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:11.150180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.193:43925)
0928 08:57:11.150220 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:11.150245 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 08:57:11.150370 8407 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 08:57:11.189960 8411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:11.190152 8411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:11.190176 8411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:11.191278 8411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:11.191309 8411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:11.191334 8411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:57:11.192682 8411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:44641
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=41501
--tserver_master_addrs=127.23.167.254:36677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:11.192857 8411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:11.193040 8411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:11.197764 8419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:11.197877 8411 server_base.cc:1034] running on GCE node
W20260928 08:57:11.197750 8417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:11.197952 8416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:11.198153 8411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:11.198354 8411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:11.199496 8411 hybrid_clock.cc:648] HybridClock initialized: now 1790585831199478 us; error 29 us; skew 500 ppm
Sep 28 08:57:11 dist-test-slave-pbjp krb5kdc[7103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585831, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:11.201838 8411 init.cc:377] Logged in from keytab as kudu/127.23.167.194@KRBTEST.COM (short username kudu)
I20260928 08:57:11.202291 8411 webserver.cc:460] Webserver started at http://127.23.167.194:41501/ using document root <none> and password file <none>
I20260928 08:57:11.202489 8411 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:11.202533 8411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:11.203845 8411 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:57:11.204421 8426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:11.204553 8411 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:57:11.204632 8411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "ffb3b609b7154fa4937dd34afe6adf7b"
format_stamp: "Formatted at 2026-09-28 08:56:29 on dist-test-slave-pbjp"
server_key: "1ee48a1a821ea765f9b6ceddbe636281"
server_key_iv: "86562fe4997eb5285122c99efe4b06f9"
server_key_version: "encryptionkey@0"
I20260928 08:57:11.204994 8411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:11.224474 8411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:11.226302 8411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:11.226428 8411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:11.226665 8411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:57:11.226974 8411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:57:11.227008 8411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:11.227037 8411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:57:11.227061 8411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:11.232795 8411 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:44641
I20260928 08:57:11.232848 8539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:44641 every 8 connection(s)
I20260928 08:57:11.233243 8411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
I20260928 08:57:11.234329 8542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:11.233922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.194:47239)
0928 08:57:11.234090 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:11.234180 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":75,"threads_started":1}
I20260928 08:57:11.234483 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 8411
I20260928 08:57:11.234736 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:36981
--local_ip_for_outbound_sockets=127.23.167.195
--tserver_master_addrs=127.23.167.254:36677
--webserver_port=45705
--webserver_interface=127.23.167.195
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:11.234771 8540 heartbeater.cc:644] Failed to heartbeat to 127.23.167.254:36677 (0 consecutive failures): Network error: Failed to ping master at 127.23.167.254:36677: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
I20260928 08:57:11.235049 8542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:11.234933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.194:39989)
0928 08:57:11.234992 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:11.235023 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 08:57:11.235318 8542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:11.235242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.194:57797)
0928 08:57:11.235279 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:11.235301 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 08:57:11.235419 8540 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 08:57:11.317895 8543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:11.318101 8543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:11.318133 8543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:11.319250 8543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:11.319288 8543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:11.319322 8543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:57:11.320741 8543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:36981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=45705
--tserver_master_addrs=127.23.167.254:36677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:11.320962 8543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:11.321199 8543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:11.325912 8548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:11.325945 8551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:11.326032 8543 server_base.cc:1034] running on GCE node
W20260928 08:57:11.325902 8549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:11.326333 8543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:11.326527 8543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:11.327662 8543 hybrid_clock.cc:648] HybridClock initialized: now 1790585831327649 us; error 27 us; skew 500 ppm
Sep 28 08:57:11 dist-test-slave-pbjp krb5kdc[7103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585831, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:11.329979 8543 init.cc:377] Logged in from keytab as kudu/127.23.167.195@KRBTEST.COM (short username kudu)
I20260928 08:57:11.330444 8543 webserver.cc:460] Webserver started at http://127.23.167.195:45705/ using document root <none> and password file <none>
I20260928 08:57:11.330644 8543 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:11.330693 8543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:11.331990 8543 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:57:11.332736 8558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:11.332888 8543 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:57:11.332969 8543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "a6b8d92ca2404ddeb0a1a52edbae0575"
format_stamp: "Formatted at 2026-09-28 08:56:29 on dist-test-slave-pbjp"
server_key: "c81c39ea847e05682a2f036b23a46744"
server_key_iv: "a9ffed92561b9e94d80b335377750023"
server_key_version: "encryptionkey@0"
I20260928 08:57:11.333369 8543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:11.348484 8543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:11.350389 8543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:11.350498 8543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:11.350687 8543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:57:11.351049 8565 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260928 08:57:11.352511 8565 ts_tablet_manager.cc:1852] T b6f55a780d074d6eb6dad0a13e0fc27a P a6b8d92ca2404ddeb0a1a52edbae0575: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 08:57:11.352775 8565 ts_tablet_manager.cc:1905] T b6f55a780d074d6eb6dad0a13e0fc27a P a6b8d92ca2404ddeb0a1a52edbae0575: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:57:11.355228 8565 ts_tablet_manager.cc:1918] T b6f55a780d074d6eb6dad0a13e0fc27a P a6b8d92ca2404ddeb0a1a52edbae0575: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:57:11.355309 8565 log.cc:1196] T b6f55a780d074d6eb6dad0a13e0fc27a P a6b8d92ca2404ddeb0a1a52edbae0575: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/wal/wals/b6f55a780d074d6eb6dad0a13e0fc27a
I20260928 08:57:11.355583 8565 ts_tablet_manager.cc:1939] T b6f55a780d074d6eb6dad0a13e0fc27a P a6b8d92ca2404ddeb0a1a52edbae0575: Deleting consensus metadata
I20260928 08:57:11.355773 8543 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 08:57:11.355824 8543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260928 08:57:11.355852 8543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:57:11.355876 8543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:11.361660 8543 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:36981
I20260928 08:57:11.361732 8672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:36981 every 8 connection(s)
I20260928 08:57:11.361997 8543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
I20260928 08:57:11.363070 8675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:11.362678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.195:51085)
0928 08:57:11.362855 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:11.362933 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":75,"threads_started":1}
W20260928 08:57:11.363276 8673 heartbeater.cc:644] Failed to heartbeat to 127.23.167.254:36677 (0 consecutive failures): Network error: Failed to ping master at 127.23.167.254:36677: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
I20260928 08:57:11.363512 8675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:11.363435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.195:40853)
0928 08:57:11.363466 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:11.363494 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260928 08:57:11.363752 8675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:11.363682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.195:57607)
0928 08:57:11.363714 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:11.363739 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 08:57:11.363837 8673 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 08:57:11.369889 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 8543
I20260928 08:57:12.151054 8677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:12.150794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.193:57211)
0928 08:57:12.150932 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:12.150979 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
I20260928 08:57:12.235960 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:12.235764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.194:51841)
0928 08:57:12.235868 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:12.235907 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":52,"threads_started":1}
I20260928 08:57:12.364404 8679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:12.364202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.195:51501)
0928 08:57:12.364331 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:12.364369 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260928 08:57:12.580801 8680 raft_consensus.cc:482] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:57:12.580919 8680 raft_consensus.cc:504] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 36981 } } peers { permanent_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 44641 } } peers { permanent_uuid: "e2c63387905b4bef8a4057433fedefca" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46519 } }
I20260928 08:57:12.581284 8680 leader_election.cc:288] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a6b8d92ca2404ddeb0a1a52edbae0575 (127.23.167.195:36981), ffb3b609b7154fa4937dd34afe6adf7b (127.23.167.194:44641)
Sep 28 08:57:12 dist-test-slave-pbjp krb5kdc[7103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585831, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for kudu/127.23.167.195@KRBTEST.COM
Sep 28 08:57:12 dist-test-slave-pbjp krb5kdc[7103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585831, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for kudu/127.23.167.194@KRBTEST.COM
I20260928 08:57:12.591055 8683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:12.581648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:42717 (local address 127.23.167.195:36981)
0928 08:57:12.581758 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:12.581766 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:12.581783 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:57:12.582254 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:12.582256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:12.582291 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:12.582407 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:12.583357 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:12.584188 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:12.585147 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:12.585539 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:12.585634 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:12.587851 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:12.587855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:12.587855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:12.587866 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:12.589523 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:12.589981 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:12.589984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:12.589985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:12.590069 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:12.590278 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:12.590279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:12.590279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:12.590503 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:12.590577 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:12.590825 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 08:57:12.590951 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":49,"threads_started":1}
I20260928 08:57:12.591055 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:12.581561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:47203 (local address 127.23.167.194:44641)
0928 08:57:12.581738 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:12.581750 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:12.581784 (+ 34us) server_negotiation.cc:407] Connection header received
0928 08:57:12.582237 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:12.582239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:12.582281 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:12.582407 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:12.583193 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:12.584116 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:12.585161 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:12.585539 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:12.585634 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:12.588197 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:12.588201 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:12.588201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:12.588211 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:12.589599 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:12.589966 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:12.589969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:12.589969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:12.590057 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:12.590274 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:12.590275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:12.590276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:12.590492 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:12.590556 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:12.590822 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 08:57:12.590933 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":63,"threads_started":1}
I20260928 08:57:12.591065 8681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:12.581447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:36981 (local address 127.23.167.193:42717)
0928 08:57:12.581602 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:12.581612 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:12.581649 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:12.582487 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:12.582489 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:12.582839 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:57:12.583250 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:12.583253 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:12.584271 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:12.584272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:12.585037 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:12.585039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:12.585618 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:12.585619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:12.585736 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:12.586276 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:57:12.586286 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:57:12.587757 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:57:12.589600 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:12.589606 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:12.589609 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:12.589900 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:12.590129 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:12.590129 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:12.590130 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:12.590221 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:12.590569 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:57:12.590569 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:57:12.590679 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 08:57:12.590762 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 08:57:12.590940 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":86,"thread_start_us":65,"threads_started":1}
I20260928 08:57:12.591306 8677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:12.581439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:44641 (local address 127.23.167.193:47203)
0928 08:57:12.581489 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:12.581502 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:12.581570 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:12.582466 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:12.582468 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:12.582838 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:57:12.583068 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:12.583075 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:12.584209 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:12.584210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:12.585039 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:12.585041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:12.585622 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:12.585623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:12.585736 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:12.586324 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:57:12.586329 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:57:12.588109 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:57:12.589674 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:12.589675 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:12.589675 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:12.589898 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:12.590116 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:12.590117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:12.590117 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:12.590219 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:12.590555 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:57:12.590562 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:57:12.590681 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 08:57:12.590763 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 08:57:12.591009 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":159}
W20260928 08:57:12.592032 8295 leader_election.cc:341] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ffb3b609b7154fa4937dd34afe6adf7b (127.23.167.194:44641): Not found: Tablet not found: b6f55a780d074d6eb6dad0a13e0fc27a
W20260928 08:57:12.592397 8294 leader_election.cc:341] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a6b8d92ca2404ddeb0a1a52edbae0575 (127.23.167.195:36981): Not found: Tablet not found: b6f55a780d074d6eb6dad0a13e0fc27a
I20260928 08:57:12.592438 8294 leader_election.cc:302] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2c63387905b4bef8a4057433fedefca; no voters: a6b8d92ca2404ddeb0a1a52edbae0575, ffb3b609b7154fa4937dd34afe6adf7b
I20260928 08:57:12.592561 8680 raft_consensus.cc:2726] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:57:13.151819 8684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:13.151556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.193:49019)
0928 08:57:13.151692 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:13.151758 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20260928 08:57:13.236531 8685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:13.236346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.194:38159)
0928 08:57:13.236450 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:13.236479 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":57,"threads_started":1}
I20260928 08:57:13.365113 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:13.364853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.195:59853)
0928 08:57:13.364981 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:13.365027 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260928 08:57:14.152581 8687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:14.152327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.193:49101)
0928 08:57:14.152479 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:14.152532 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":68,"threads_started":1}
I20260928 08:57:14.209519 8688 raft_consensus.cc:482] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:57:14.209622 8688 raft_consensus.cc:504] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 36981 } } peers { permanent_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 44641 } } peers { permanent_uuid: "e2c63387905b4bef8a4057433fedefca" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46519 } }
I20260928 08:57:14.209765 8688 leader_election.cc:288] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a6b8d92ca2404ddeb0a1a52edbae0575 (127.23.167.195:36981), ffb3b609b7154fa4937dd34afe6adf7b (127.23.167.194:44641)
W20260928 08:57:14.210182 8295 leader_election.cc:341] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ffb3b609b7154fa4937dd34afe6adf7b (127.23.167.194:44641): Not found: Tablet not found: b6f55a780d074d6eb6dad0a13e0fc27a
W20260928 08:57:14.210253 8294 leader_election.cc:341] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a6b8d92ca2404ddeb0a1a52edbae0575 (127.23.167.195:36981): Not found: Tablet not found: b6f55a780d074d6eb6dad0a13e0fc27a
I20260928 08:57:14.210340 8294 leader_election.cc:302] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2c63387905b4bef8a4057433fedefca; no voters: a6b8d92ca2404ddeb0a1a52edbae0575, ffb3b609b7154fa4937dd34afe6adf7b
I20260928 08:57:14.210428 8688 raft_consensus.cc:2726] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:57:14.237161 8689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:14.236909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.194:46979)
0928 08:57:14.237052 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:14.237112 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":58,"threads_started":1}
I20260928 08:57:14.365738 8690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:14.365508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.195:56721)
0928 08:57:14.365657 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:14.365697 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":73,"threads_started":1}
I20260928 08:57:15.153484 8691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:15.153199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.193:48947)
0928 08:57:15.153369 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:15.153425 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":72,"threads_started":1}
I20260928 08:57:15.237848 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:15.237614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.194:33743)
0928 08:57:15.237755 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:15.237799 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260928 08:57:15.366412 8693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:15.366200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.195:46251)
0928 08:57:15.366333 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:15.366369 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
I20260928 08:57:16.154338 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:16.154099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.193:60537)
0928 08:57:16.154240 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:16.154289 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
I20260928 08:57:16.238524 8695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:16.238306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.194:53145)
0928 08:57:16.238436 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:16.238481 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
I20260928 08:57:16.277706 8696 raft_consensus.cc:482] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:57:16.277814 8696 raft_consensus.cc:504] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 36981 } } peers { permanent_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 44641 } } peers { permanent_uuid: "e2c63387905b4bef8a4057433fedefca" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46519 } }
I20260928 08:57:16.277979 8696 leader_election.cc:288] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a6b8d92ca2404ddeb0a1a52edbae0575 (127.23.167.195:36981), ffb3b609b7154fa4937dd34afe6adf7b (127.23.167.194:44641)
W20260928 08:57:16.278348 8295 leader_election.cc:341] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ffb3b609b7154fa4937dd34afe6adf7b (127.23.167.194:44641): Not found: Tablet not found: b6f55a780d074d6eb6dad0a13e0fc27a
W20260928 08:57:16.278450 8294 leader_election.cc:341] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a6b8d92ca2404ddeb0a1a52edbae0575 (127.23.167.195:36981): Not found: Tablet not found: b6f55a780d074d6eb6dad0a13e0fc27a
I20260928 08:57:16.278510 8294 leader_election.cc:302] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2c63387905b4bef8a4057433fedefca; no voters: a6b8d92ca2404ddeb0a1a52edbae0575, ffb3b609b7154fa4937dd34afe6adf7b
I20260928 08:57:16.278594 8696 raft_consensus.cc:2726] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:57:16.367123 8697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:16.366870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.195:54405)
0928 08:57:16.367036 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:16.367076 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":86,"threads_started":1}
I20260928 08:57:17.155131 8698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:17.154858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.193:36561)
0928 08:57:17.155024 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:17.155075 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":80,"threads_started":1}
I20260928 08:57:17.239187 8699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:17.238978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.194:48959)
0928 08:57:17.239104 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:17.239139 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260928 08:57:17.367722 8700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:17.367511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.195:36571)
0928 08:57:17.367643 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:17.367679 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
I20260928 08:57:18.155915 8701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:18.155658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.193:35025)
0928 08:57:18.155820 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:18.155867 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":63,"threads_started":1}
I20260928 08:57:18.239886 8702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:18.239637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.194:41481)
0928 08:57:18.239802 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:18.239841 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":97,"threads_started":1}
I20260928 08:57:18.368364 8703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:18.368145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.195:60011)
0928 08:57:18.368273 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:18.368316 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260928 08:57:19.156757 8704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:19.156505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.193:33417)
0928 08:57:19.156655 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:19.156706 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":70,"threads_started":1}
I20260928 08:57:19.240604 8705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:19.240348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.194:49355)
0928 08:57:19.240506 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:19.240554 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":64,"threads_started":1}
I20260928 08:57:19.369095 8706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:19.368831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.195:51325)
0928 08:57:19.368967 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:19.369013 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260928 08:57:20.157577 8707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:20.157300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.193:51813)
0928 08:57:20.157465 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:20.157520 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":66,"threads_started":1}
I20260928 08:57:20.191059 8708 raft_consensus.cc:482] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:57:20.191147 8708 raft_consensus.cc:504] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 36981 } } peers { permanent_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 44641 } } peers { permanent_uuid: "e2c63387905b4bef8a4057433fedefca" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46519 } }
I20260928 08:57:20.191313 8708 leader_election.cc:288] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a6b8d92ca2404ddeb0a1a52edbae0575 (127.23.167.195:36981), ffb3b609b7154fa4937dd34afe6adf7b (127.23.167.194:44641)
W20260928 08:57:20.191681 8294 leader_election.cc:341] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a6b8d92ca2404ddeb0a1a52edbae0575 (127.23.167.195:36981): Not found: Tablet not found: b6f55a780d074d6eb6dad0a13e0fc27a
W20260928 08:57:20.191761 8295 leader_election.cc:341] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ffb3b609b7154fa4937dd34afe6adf7b (127.23.167.194:44641): Not found: Tablet not found: b6f55a780d074d6eb6dad0a13e0fc27a
I20260928 08:57:20.191833 8295 leader_election.cc:302] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2c63387905b4bef8a4057433fedefca; no voters: a6b8d92ca2404ddeb0a1a52edbae0575, ffb3b609b7154fa4937dd34afe6adf7b
I20260928 08:57:20.191917 8708 raft_consensus.cc:2726] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:57:20.241408 8709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:20.241165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.194:55485)
0928 08:57:20.241301 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:20.241346 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20260928 08:57:20.369760 8710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:20.369539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.195:43371)
0928 08:57:20.369664 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:20.369708 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":57,"threads_started":1}
I20260928 08:57:21.158392 8711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:21.158152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.193:45583)
0928 08:57:21.158298 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:21.158343 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":65,"threads_started":1}
I20260928 08:57:21.242095 8712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:21.241864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.194:53845)
0928 08:57:21.241986 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:21.242030 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20260928 08:57:21.370416 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:21.370194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.195:38285)
0928 08:57:21.370334 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:21.370374 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":82,"threads_started":1}
I20260928 08:57:22.159224 8714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:22.158969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.193:43589)
0928 08:57:22.159118 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:22.159172 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
I20260928 08:57:22.242719 8715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:22.242502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.194:42499)
0928 08:57:22.242639 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:22.242671 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
I20260928 08:57:22.371075 8716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:22.370847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.195:44213)
0928 08:57:22.370989 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:22.371029 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260928 08:57:23.159950 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:23.159730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.193:58051)
0928 08:57:23.159866 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:23.159903 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":66,"threads_started":1}
I20260928 08:57:23.243414 8718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:23.243184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.194:59107)
0928 08:57:23.243315 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:23.243361 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
I20260928 08:57:23.371790 8719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:23.371551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.195:37349)
0928 08:57:23.371701 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:23.371742 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20260928 08:57:24.160778 8720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:24.160490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.193:42223)
0928 08:57:24.160677 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:24.160728 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":94,"threads_started":1}
I20260928 08:57:24.244156 8721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:24.243926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.194:57177)
0928 08:57:24.244062 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:24.244108 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260928 08:57:24.372561 8722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:24.372306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.195:46921)
0928 08:57:24.372462 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:24.372506 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":82,"threads_started":1}
I20260928 08:57:25.161476 8723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:25.161241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.193:55075)
0928 08:57:25.161394 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:25.161428 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":94,"threads_started":1}
I20260928 08:57:25.244907 8724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:25.244689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.194:54763)
0928 08:57:25.244820 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:25.244853 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260928 08:57:25.373207 8725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:25.372979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.195:49151)
0928 08:57:25.373125 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:25.373161 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":63,"threads_started":1}
I20260928 08:57:25.724409 8726 raft_consensus.cc:482] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:57:25.724515 8726 raft_consensus.cc:504] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b8d92ca2404ddeb0a1a52edbae0575" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 36981 } } peers { permanent_uuid: "ffb3b609b7154fa4937dd34afe6adf7b" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 44641 } } peers { permanent_uuid: "e2c63387905b4bef8a4057433fedefca" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 46519 } }
I20260928 08:57:25.724661 8726 leader_election.cc:288] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a6b8d92ca2404ddeb0a1a52edbae0575 (127.23.167.195:36981), ffb3b609b7154fa4937dd34afe6adf7b (127.23.167.194:44641)
W20260928 08:57:25.725024 8294 leader_election.cc:341] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a6b8d92ca2404ddeb0a1a52edbae0575 (127.23.167.195:36981): Not found: Tablet not found: b6f55a780d074d6eb6dad0a13e0fc27a
W20260928 08:57:25.725142 8295 leader_election.cc:341] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ffb3b609b7154fa4937dd34afe6adf7b (127.23.167.194:44641): Not found: Tablet not found: b6f55a780d074d6eb6dad0a13e0fc27a
I20260928 08:57:25.725210 8295 leader_election.cc:302] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2c63387905b4bef8a4057433fedefca; no voters: a6b8d92ca2404ddeb0a1a52edbae0575, ffb3b609b7154fa4937dd34afe6adf7b
I20260928 08:57:25.725289 8726 raft_consensus.cc:2726] T b6f55a780d074d6eb6dad0a13e0fc27a P e2c63387905b4bef8a4057433fedefca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:57:26.162281 8727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:26.162030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.193:51099)
0928 08:57:26.162186 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:26.162232 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":68,"threads_started":1}
I20260928 08:57:26.245671 8728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:26.245398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.194:57161)
0928 08:57:26.245581 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:26.245622 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":98,"threads_started":1}
I20260928 08:57:26.373976 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:26.373753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:36677 (local address 127.23.167.195:55107)
0928 08:57:26.373895 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:26.373931 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.254:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":65,"threads_started":1}
I20260928 08:57:26.379763 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 8242
I20260928 08:57:26.383978 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 8411
I20260928 08:57:26.387672 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 8543
2026-09-28T08:57:26Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58272 ms)
[----------] 30 tests from SecurityITest (256394 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:57:26 dist-test-slave-pbjp krb5kdc[8732](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:57:26 dist-test-slave-pbjp krb5kdc[8732](info): set up 2 sockets
Sep 28 08:57:26 dist-test-slave-pbjp krb5kdc[8732](info): commencing operation
krb5kdc: starting...
W20260928 08:57:27.340603 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:57:27 dist-test-slave-pbjp krb5kdc[8732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:57:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:57:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.167.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
I20260928 08:57:27.451920 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:41781
--webserver_interface=127.23.167.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:46295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:41781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:27.530746 8745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:27.530954 8745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:27.530978 8745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:27.532267 8745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:57:27.532311 8745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:27.532325 8745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:27.532336 8745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:57:27.532354 8745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:57:27.533855 8745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:46295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:41781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:41781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:27.534081 8745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:27.534282 8745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:27.539345 8750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:27.539389 8745 server_base.cc:1034] running on GCE node
W20260928 08:57:27.539348 8753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:27.539301 8751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:27.539744 8745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:27.540015 8745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:27.541115 8745 hybrid_clock.cc:648] HybridClock initialized: now 1790585847541100 us; error 25 us; skew 500 ppm
Sep 28 08:57:27 dist-test-slave-pbjp krb5kdc[8732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585847, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:27.543394 8745 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:57:27.543826 8745 webserver.cc:460] Webserver started at http://127.23.167.254:33165/ using document root <none> and password file <none>
I20260928 08:57:27.544016 8745 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:27.544062 8745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:27.544181 8745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:27.545223 8745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "bf735217cf704392a205fda76e244d40"
format_stamp: "Formatted at 2026-09-28 08:57:27 on dist-test-slave-pbjp"
server_key: "8078871f4371d056bef9a5283e393a16"
server_key_iv: "7a43518eb69694752253adca7e07b7fe"
server_key_version: "encryptionkey@0"
I20260928 08:57:27.545554 8745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "bf735217cf704392a205fda76e244d40"
format_stamp: "Formatted at 2026-09-28 08:57:27 on dist-test-slave-pbjp"
server_key: "8078871f4371d056bef9a5283e393a16"
server_key_iv: "7a43518eb69694752253adca7e07b7fe"
server_key_version: "encryptionkey@0"
I20260928 08:57:27.546763 8745 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:57:27.547425 8760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:27.547580 8745 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:57:27.547662 8745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "bf735217cf704392a205fda76e244d40"
format_stamp: "Formatted at 2026-09-28 08:57:27 on dist-test-slave-pbjp"
server_key: "8078871f4371d056bef9a5283e393a16"
server_key_iv: "7a43518eb69694752253adca7e07b7fe"
server_key_version: "encryptionkey@0"
I20260928 08:57:27.547719 8745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:27.563961 8745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:27.565982 8745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:27.566114 8745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:27.569844 8745 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:41781
I20260928 08:57:27.569887 8812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:41781 every 8 connection(s)
I20260928 08:57:27.570202 8745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:57:27.571007 8813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:27.573796 8813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf735217cf704392a205fda76e244d40: Bootstrap starting.
I20260928 08:57:27.574457 8813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf735217cf704392a205fda76e244d40: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:27.574751 8813 log.cc:824] T 00000000000000000000000000000000 P bf735217cf704392a205fda76e244d40: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:27.575387 8813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf735217cf704392a205fda76e244d40: No bootstrap required, opened a new log
I20260928 08:57:27.576246 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 8745
I20260928 08:57:27.576334 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:57:27.576505 24223 external_mini_cluster.cc:1428] Setting key aa52ad35695bfa7c94d38f021413103c
I20260928 08:57:27.577273 8813 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf735217cf704392a205fda76e244d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf735217cf704392a205fda76e244d40" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 41781 } }
I20260928 08:57:27.577442 8813 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf735217cf704392a205fda76e244d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:27.577486 8813 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf735217cf704392a205fda76e244d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf735217cf704392a205fda76e244d40, State: Initialized, Role: FOLLOWER
I20260928 08:57:27.577612 8813 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf735217cf704392a205fda76e244d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf735217cf704392a205fda76e244d40" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 41781 } }
I20260928 08:57:27.577690 8813 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf735217cf704392a205fda76e244d40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:57:27.577728 8813 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf735217cf704392a205fda76e244d40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:57:27.577775 8813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf735217cf704392a205fda76e244d40 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:27.578737 8813 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf735217cf704392a205fda76e244d40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf735217cf704392a205fda76e244d40" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 41781 } }
I20260928 08:57:27.578826 8813 leader_election.cc:302] T 00000000000000000000000000000000 P bf735217cf704392a205fda76e244d40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf735217cf704392a205fda76e244d40; no voters:
I20260928 08:57:27.578980 8813 leader_election.cc:288] T 00000000000000000000000000000000 P bf735217cf704392a205fda76e244d40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:57:27.579022 8818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf735217cf704392a205fda76e244d40 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:57:27.579206 8818 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf735217cf704392a205fda76e244d40 [term 1 LEADER]: Becoming Leader. State: Replica: bf735217cf704392a205fda76e244d40, State: Running, Role: LEADER
I20260928 08:57:27.579222 8813 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf735217cf704392a205fda76e244d40 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:57:27.579349 8818 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf735217cf704392a205fda76e244d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf735217cf704392a205fda76e244d40" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 41781 } }
I20260928 08:57:27.579715 8819 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf735217cf704392a205fda76e244d40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf735217cf704392a205fda76e244d40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf735217cf704392a205fda76e244d40" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 41781 } } }
I20260928 08:57:27.579802 8819 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf735217cf704392a205fda76e244d40 [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:27.579945 8820 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf735217cf704392a205fda76e244d40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf735217cf704392a205fda76e244d40. Latest consensus state: current_term: 1 leader_uuid: "bf735217cf704392a205fda76e244d40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf735217cf704392a205fda76e244d40" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 41781 } } }
I20260928 08:57:27.580011 8820 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf735217cf704392a205fda76e244d40 [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:27.580116 8827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:57:27.580554 8827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:57:27.582278 8827 catalog_manager.cc:1347] Generated new cluster ID: 4a1233e9c6304523b7652003a5663732
I20260928 08:57:27.582325 8827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 08:57:27 dist-test-slave-pbjp krb5kdc[8732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:57:27.588261 8817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:27.576986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46236 (local address 127.23.167.254:41781)
0928 08:57:27.577187 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:27.577197 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:27.577230 (+ 33us) server_negotiation.cc:407] Connection header received
0928 08:57:27.577344 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:27.577351 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:27.577407 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:27.577568 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:27.578420 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:27.579604 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:27.580698 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:27.581301 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:27.581386 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:27.584227 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:27.584231 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:27.584231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:27.584243 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:27.586207 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:27.586600 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:27.586603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:27.586603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:27.586742 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:27.586939 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:27.586939 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:27.586940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:27.587246 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:27.587433 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:27.587921 (+ 488us) server_negotiation.cc:299] Negotiation successful
0928 08:57:27.588061 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":108,"threads_started":1}
I20260928 08:57:27.596379 8827 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:57:27.596863 8827 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:57:27.599566 8827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf735217cf704392a205fda76e244d40: Generated new TSK 0
I20260928 08:57:27.599714 8827 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.167.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
I20260928 08:57:27.625208 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:0
--local_ip_for_outbound_sockets=127.23.167.193
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:41781
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:46295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:27.704869 8839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:27.705107 8839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:27.705132 8839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:27.706310 8839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:27.706352 8839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:27.706380 8839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:57:27.707883 8839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:46295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:41781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:27.708086 8839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:27.708271 8839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:27.713352 8845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:27.713392 8847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:27.713449 8839 server_base.cc:1034] running on GCE node
W20260928 08:57:27.713409 8844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:27.713729 8839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:27.713982 8839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:27.715157 8839 hybrid_clock.cc:648] HybridClock initialized: now 1790585847715139 us; error 32 us; skew 500 ppm
Sep 28 08:57:27 dist-test-slave-pbjp krb5kdc[8732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585847, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:27.717734 8839 init.cc:377] Logged in from keytab as kudu/127.23.167.193@KRBTEST.COM (short username kudu)
I20260928 08:57:27.718181 8839 webserver.cc:460] Webserver started at http://127.23.167.193:44367/ using document root <none> and password file <none>
I20260928 08:57:27.718410 8839 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:27.718468 8839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:27.718595 8839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:27.719707 8839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-0/data/instance:
uuid: "f59ef63dbf234699bb291e0b2c1db841"
format_stamp: "Formatted at 2026-09-28 08:57:27 on dist-test-slave-pbjp"
server_key: "5faa8d270804d04994739204ad75abf1"
server_key_iv: "41d08c99a36470e77c77d3e7244a775c"
server_key_version: "encryptionkey@0"
I20260928 08:57:27.720048 8839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance:
uuid: "f59ef63dbf234699bb291e0b2c1db841"
format_stamp: "Formatted at 2026-09-28 08:57:27 on dist-test-slave-pbjp"
server_key: "5faa8d270804d04994739204ad75abf1"
server_key_iv: "41d08c99a36470e77c77d3e7244a775c"
server_key_version: "encryptionkey@0"
I20260928 08:57:27.721315 8839 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:57:27.722025 8854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:27.722168 8839 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:57:27.722245 8839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "f59ef63dbf234699bb291e0b2c1db841"
format_stamp: "Formatted at 2026-09-28 08:57:27 on dist-test-slave-pbjp"
server_key: "5faa8d270804d04994739204ad75abf1"
server_key_iv: "41d08c99a36470e77c77d3e7244a775c"
server_key_version: "encryptionkey@0"
I20260928 08:57:27.722304 8839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:27.758198 8839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:27.760388 8839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:27.760528 8839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:27.760764 8839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:57:27.761093 8839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:57:27.761126 8839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:27.761155 8839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:57:27.761173 8839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:27.766723 8839 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:37767
I20260928 08:57:27.766778 8967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:37767 every 8 connection(s)
I20260928 08:57:27.767096 8839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
I20260928 08:57:27.769927 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 8839
I20260928 08:57:27.770032 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance
I20260928 08:57:27.770234 24223 external_mini_cluster.cc:1428] Setting key 7580a70d222efa63be59b82e875f81db
Sep 28 08:57:27 dist-test-slave-pbjp krb5kdc[8732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585847, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:57:27.776400 8817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:27.767823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:34183 (local address 127.23.167.254:41781)
0928 08:57:27.767870 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:27.767872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:27.768076 (+ 204us) server_negotiation.cc:407] Connection header received
0928 08:57:27.768568 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:27.768569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:27.768583 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:27.768648 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:27.769629 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:27.770372 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:27.771561 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:27.771854 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:27.771922 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:27.773942 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:27.773943 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:27.773943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:27.773952 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:27.775189 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:27.775541 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:27.775542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:27.775542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:27.775595 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:27.775804 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:27.775805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:27.775805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:27.775910 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:27.775965 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:27.776225 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 08:57:27.776312 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 08:57:27.776554 8970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:27.767788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:41781 (local address 127.23.167.193:34183)
0928 08:57:27.767952 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:27.767996 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:27.768469 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:27.768728 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:27.768732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:27.769317 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:57:27.769537 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:27.769542 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:27.770531 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:27.770533 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:27.771463 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:27.771466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:27.771933 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:27.771934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:27.772060 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:27.772556 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:57:27.772564 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:57:27.773857 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:57:27.775262 (+ 1405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:27.775270 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:27.775272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:27.775476 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:27.775646 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:27.775647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:27.775647 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:27.775751 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:27.775963 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:57:27.775970 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:57:27.776109 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 08:57:27.776194 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 08:57:27.776343 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":70,"threads_started":1}
I20260928 08:57:27.777015 8968 heartbeater.cc:347] Connected to a master server at 127.23.167.254:41781
I20260928 08:57:27.777125 8968 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.23.167.194@KRBTEST.COM; defaulting to no policy
I20260928 08:57:27.777302 8968 heartbeater.cc:511] Master 127.23.167.254:41781 requested a full tablet report, sending...
I20260928 08:57:27.777802 8777 ts_manager.cc:194] Registered new tserver with Master: f59ef63dbf234699bb291e0b2c1db841 (127.23.167.193:37767)
I20260928 08:57:27.778506 8777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.193@KRBTEST.COM'} at 127.23.167.193:34183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
I20260928 08:57:27.793807 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:0
--local_ip_for_outbound_sockets=127.23.167.194
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:41781
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:46295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:27.872882 8973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:27.873116 8973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:27.873149 8973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:27.874289 8973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:27.874326 8973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:27.874359 8973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:57:27.875751 8973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:46295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:41781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:27.875985 8973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:27.876196 8973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:27.880890 8981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:27.880914 8979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:27.880894 8978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:27.881127 8973 server_base.cc:1034] running on GCE node
I20260928 08:57:27.881306 8973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:27.881541 8973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:27.882688 8973 hybrid_clock.cc:648] HybridClock initialized: now 1790585847882671 us; error 27 us; skew 500 ppm
Sep 28 08:57:27 dist-test-slave-pbjp krb5kdc[8732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585847, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:27.885174 8973 init.cc:377] Logged in from keytab as kudu/127.23.167.194@KRBTEST.COM (short username kudu)
I20260928 08:57:27.885589 8973 webserver.cc:460] Webserver started at http://127.23.167.194:37029/ using document root <none> and password file <none>
I20260928 08:57:27.885804 8973 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:27.885850 8973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:27.885973 8973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:27.886952 8973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-1/data/instance:
uuid: "351df5aa96a340afa40d099de3983611"
format_stamp: "Formatted at 2026-09-28 08:57:27 on dist-test-slave-pbjp"
server_key: "b5509c634519bd910b4f577574bbf45f"
server_key_iv: "f92c0be8b9d7d2bec034240b6fee290b"
server_key_version: "encryptionkey@0"
I20260928 08:57:27.887267 8973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance:
uuid: "351df5aa96a340afa40d099de3983611"
format_stamp: "Formatted at 2026-09-28 08:57:27 on dist-test-slave-pbjp"
server_key: "b5509c634519bd910b4f577574bbf45f"
server_key_iv: "f92c0be8b9d7d2bec034240b6fee290b"
server_key_version: "encryptionkey@0"
I20260928 08:57:27.888518 8973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:57:27.889209 8988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:27.889356 8973 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:27.889464 8973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "351df5aa96a340afa40d099de3983611"
format_stamp: "Formatted at 2026-09-28 08:57:27 on dist-test-slave-pbjp"
server_key: "b5509c634519bd910b4f577574bbf45f"
server_key_iv: "f92c0be8b9d7d2bec034240b6fee290b"
server_key_version: "encryptionkey@0"
I20260928 08:57:27.889526 8973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:27.901211 8973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:27.903090 8973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:27.903216 8973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:27.903414 8973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:57:27.903710 8973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:57:27.903741 8973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:27.903769 8973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:57:27.903792 8973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:27.909230 8973 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:34901
I20260928 08:57:27.909286 9101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:34901 every 8 connection(s)
I20260928 08:57:27.909612 8973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
Sep 28 08:57:27 dist-test-slave-pbjp krb5kdc[8732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585847, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:57:27.918074 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 8973
I20260928 08:57:27.918190 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance
I20260928 08:57:27.918309 8817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:27.910312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:59561 (local address 127.23.167.254:41781)
0928 08:57:27.910378 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:27.910380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:27.910554 (+ 174us) server_negotiation.cc:407] Connection header received
0928 08:57:27.911091 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:27.911092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:27.911103 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:27.911156 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:27.912073 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:27.912579 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:27.913638 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:27.913833 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:27.913887 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:27.915921 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:27.915922 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:27.915923 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:27.915929 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:27.917198 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:27.917518 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:27.917519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:27.917519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:27.917566 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:27.917741 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:27.917742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:27.917742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:27.917838 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:27.917894 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:27.918131 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 08:57:27.918222 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 08:57:27.918438 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:27.910266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:41781 (local address 127.23.167.194:59561)
0928 08:57:27.910424 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:27.910474 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:27.910981 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:27.911213 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:27.911216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:27.911759 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:57:27.911972 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:27.911978 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:27.912657 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:27.912658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:27.913562 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:27.913565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:27.913901 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:27.913902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:27.914004 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:27.914509 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:57:27.914517 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:57:27.915841 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:57:27.917253 (+ 1412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:27.917257 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:27.917259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:27.917465 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:27.917609 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:27.917609 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:27.917610 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:27.917700 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:27.917889 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:57:27.917892 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:57:27.918011 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 08:57:27.918091 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 08:57:27.918242 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":75,"threads_started":1}
I20260928 08:57:27.918457 24223 external_mini_cluster.cc:1428] Setting key 9f7ab6496f3397bb21657d5f5e91de75
I20260928 08:57:27.919092 9102 heartbeater.cc:347] Connected to a master server at 127.23.167.254:41781
I20260928 08:57:27.919188 9102 heartbeater.cc:464] Registering TS with master...
I20260928 08:57:27.919384 9102 heartbeater.cc:511] Master 127.23.167.254:41781 requested a full tablet report, sending...
I20260928 08:57:27.919770 8776 ts_manager.cc:194] Registered new tserver with Master: 351df5aa96a340afa40d099de3983611 (127.23.167.194:34901)
I20260928 08:57:27.920135 8776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.194@KRBTEST.COM'} at 127.23.167.194:59561
WARNING: no policy specified for kudu/127.23.167.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
I20260928 08:57:27.943455 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:0
--local_ip_for_outbound_sockets=127.23.167.195
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:41781
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:46295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:28.023495 9107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:28.023693 9107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:28.023715 9107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:28.024875 9107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:28.024916 9107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:28.024941 9107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:57:28.026404 9107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:46295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:41781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:28.026597 9107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:28.026768 9107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:28.031527 9115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:28.031571 9112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:28.031635 9107 server_base.cc:1034] running on GCE node
W20260928 08:57:28.031545 9113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:28.031898 9107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:28.032130 9107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:28.033265 9107 hybrid_clock.cc:648] HybridClock initialized: now 1790585848033249 us; error 27 us; skew 500 ppm
Sep 28 08:57:28 dist-test-slave-pbjp krb5kdc[8732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585848, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:28.035538 9107 init.cc:377] Logged in from keytab as kudu/127.23.167.195@KRBTEST.COM (short username kudu)
I20260928 08:57:28.035953 9107 webserver.cc:460] Webserver started at http://127.23.167.195:40593/ using document root <none> and password file <none>
I20260928 08:57:28.036168 9107 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:28.036217 9107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:28.036337 9107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:28.037384 9107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-2/data/instance:
uuid: "75873050a9a6454b870709a7a20a8401"
format_stamp: "Formatted at 2026-09-28 08:57:28 on dist-test-slave-pbjp"
server_key: "a241e99ffa1375eaa17e7b05235ed194"
server_key_iv: "faef58f8f3199c73da7145966bbcdcc6"
server_key_version: "encryptionkey@0"
I20260928 08:57:28.037729 9107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance:
uuid: "75873050a9a6454b870709a7a20a8401"
format_stamp: "Formatted at 2026-09-28 08:57:28 on dist-test-slave-pbjp"
server_key: "a241e99ffa1375eaa17e7b05235ed194"
server_key_iv: "faef58f8f3199c73da7145966bbcdcc6"
server_key_version: "encryptionkey@0"
I20260928 08:57:28.038926 9107 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:57:28.039696 9122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:28.039861 9107 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:57:28.039938 9107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "75873050a9a6454b870709a7a20a8401"
format_stamp: "Formatted at 2026-09-28 08:57:28 on dist-test-slave-pbjp"
server_key: "a241e99ffa1375eaa17e7b05235ed194"
server_key_iv: "faef58f8f3199c73da7145966bbcdcc6"
server_key_version: "encryptionkey@0"
I20260928 08:57:28.040011 9107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:28.050741 9107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:28.052372 9107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:28.052496 9107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:28.052691 9107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:57:28.052990 9107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:57:28.053025 9107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:28.053053 9107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:57:28.053098 9107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:28.058499 9107 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:33385
I20260928 08:57:28.058549 9235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:33385 every 8 connection(s)
I20260928 08:57:28.058818 9107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
Sep 28 08:57:28 dist-test-slave-pbjp krb5kdc[8732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585848, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:57:28.067202 8817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.059441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:43997 (local address 127.23.167.254:41781)
0928 08:57:28.059527 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:28.059529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:28.059670 (+ 141us) server_negotiation.cc:407] Connection header received
0928 08:57:28.060127 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:28.060128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:28.060137 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:28.060199 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:28.061103 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.061602 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.062535 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.062709 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.062749 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.064743 (+ 1994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:28.064745 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:28.064745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:28.064751 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:28.066001 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:28.066333 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:28.066334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:28.066334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:28.066378 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:28.066654 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:28.066655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:28.066655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:28.066773 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:28.066834 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:28.067072 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 08:57:28.067139 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 08:57:28.067365 9238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.059410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:41781 (local address 127.23.167.195:43997)
0928 08:57:28.059558 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:28.059596 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:28.060052 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:28.060258 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:28.060261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:28.060815 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:57:28.061015 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:28.061020 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.061667 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.061667 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:28.062475 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:28.062477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.062760 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.062760 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:28.062851 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.063304 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:57:28.063312 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:57:28.064662 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:57:28.066070 (+ 1408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:28.066075 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:28.066077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:28.066278 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:28.066451 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:28.066452 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:28.066453 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:28.066563 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:28.066835 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:57:28.066840 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:57:28.066957 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 08:57:28.067040 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 08:57:28.067159 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260928 08:57:28.067838 9236 heartbeater.cc:347] Connected to a master server at 127.23.167.254:41781
I20260928 08:57:28.067934 9236 heartbeater.cc:464] Registering TS with master...
I20260928 08:57:28.068101 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 9107
I20260928 08:57:28.068128 9236 heartbeater.cc:511] Master 127.23.167.254:41781 requested a full tablet report, sending...
I20260928 08:57:28.068202 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance
I20260928 08:57:28.068404 24223 external_mini_cluster.cc:1428] Setting key 886bc3b5d0395fc08b54512f0974fbbe
I20260928 08:57:28.068502 8776 ts_manager.cc:194] Registered new tserver with Master: 75873050a9a6454b870709a7a20a8401 (127.23.167.195:33385)
I20260928 08:57:28.068941 8776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.195@KRBTEST.COM'} at 127.23.167.195:43997
I20260928 08:57:28.069386 24223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:57:28.075802 8817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.070638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46250 (local address 127.23.167.254:41781)
0928 08:57:28.070680 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:28.070682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:28.070770 (+ 88us) server_negotiation.cc:407] Connection header received
0928 08:57:28.070831 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:28.070832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:28.070841 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:28.070885 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:28.071424 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.071883 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.072575 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.072769 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.072820 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.073852 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:28.073853 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:28.073853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:28.073860 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:28.074822 (+ 962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:28.075109 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:28.075110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:28.075110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:28.075160 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:28.075314 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:28.075315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:28.075315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:28.075468 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:28.075553 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:28.075652 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 08:57:28.075720 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 08:57:28.077085 8776 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:57:28.078579 8776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:57:28.084477 9246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.080780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:37767 (local address 127.0.0.1:33310)
0928 08:57:28.080906 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:28.080916 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:28.080955 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:28.081423 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:28.081424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:28.081427 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:57:28.081471 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:28.081472 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.082331 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.082332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:28.083269 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:28.083271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.084198 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.084199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:28.084278 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.084292 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 08:57:28.084359 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 08:57:28.084409 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":35,"threads_started":1}
I20260928 08:57:28.084510 9245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.080670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48842 (local address 127.23.167.195:33385)
0928 08:57:28.080842 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:28.080846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:28.080856 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:57:28.080921 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:28.080924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:28.080941 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:28.080994 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:57:28.081470 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.082575 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.083668 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.084151 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.084221 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.084268 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:28.084389 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 08:57:28.084443 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
I20260928 08:57:28.084510 9248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.081055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33310 (local address 127.23.167.193:37767)
0928 08:57:28.081246 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:28.081251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:28.081260 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:57:28.081300 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:28.081302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:28.081318 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:28.081420 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:57:28.081548 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.082241 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.083362 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.084064 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.084123 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.084170 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:28.084384 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 08:57:28.084427 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":80,"threads_started":1}
I20260928 08:57:28.084678 9244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.080610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:33385 (local address 127.0.0.1:48842)
0928 08:57:28.080722 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:28.080766 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:28.080832 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:28.081320 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:28.081324 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:28.081326 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:57:28.081390 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:28.081392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.082683 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.082684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:28.083602 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:28.083604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.084223 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.084223 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:28.084292 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.084294 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 08:57:28.084577 (+ 283us) client_negotiation.cc:240] Negotiation successful
0928 08:57:28.084618 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":7,"thread_start_us":51,"threads_started":1}
I20260928 08:57:28.085040 9247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.080885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:34901 (local address 127.0.0.1:50694)
0928 08:57:28.081137 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:28.081146 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:28.081195 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:28.081760 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:28.081761 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:28.081763 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:57:28.081797 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:28.081799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.082982 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.082982 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:28.084029 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:28.084032 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.084831 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.084831 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:28.084890 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.084894 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:57:28.084946 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 08:57:28.084981 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":40,"threads_started":1}
I20260928 08:57:28.085096 9249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.081140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50694 (local address 127.23.167.194:34901)
0928 08:57:28.081550 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:28.081554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:28.081563 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:57:28.081599 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:28.081601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:28.081614 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:28.081685 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:57:28.081864 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.082885 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.084115 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.084761 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.084817 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.084847 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:28.084966 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 08:57:28.085000 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":73,"threads_started":1}
I20260928 08:57:28.085741 9170 tablet_service.cc:1467] Processing CreateTablet for tablet ac7997f0dca54703af4ae3043094a888 (DEFAULT_TABLE table=test-table [id=d79ac7d7216c40ab9d84517c696ebadf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:57:28.085721 9036 tablet_service.cc:1467] Processing CreateTablet for tablet ac7997f0dca54703af4ae3043094a888 (DEFAULT_TABLE table=test-table [id=d79ac7d7216c40ab9d84517c696ebadf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:57:28.085721 8902 tablet_service.cc:1467] Processing CreateTablet for tablet ac7997f0dca54703af4ae3043094a888 (DEFAULT_TABLE table=test-table [id=d79ac7d7216c40ab9d84517c696ebadf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:57:28.086033 9170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac7997f0dca54703af4ae3043094a888. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:28.086047 8902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac7997f0dca54703af4ae3043094a888. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:28.086047 9036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac7997f0dca54703af4ae3043094a888. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:28.089228 9252 tablet_bootstrap.cc:492] T ac7997f0dca54703af4ae3043094a888 P 351df5aa96a340afa40d099de3983611: Bootstrap starting.
I20260928 08:57:28.089262 9251 tablet_bootstrap.cc:492] T ac7997f0dca54703af4ae3043094a888 P f59ef63dbf234699bb291e0b2c1db841: Bootstrap starting.
I20260928 08:57:28.089520 9250 tablet_bootstrap.cc:492] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401: Bootstrap starting.
I20260928 08:57:28.089890 9251 tablet_bootstrap.cc:654] T ac7997f0dca54703af4ae3043094a888 P f59ef63dbf234699bb291e0b2c1db841: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:28.089900 9252 tablet_bootstrap.cc:654] T ac7997f0dca54703af4ae3043094a888 P 351df5aa96a340afa40d099de3983611: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:28.090070 9250 tablet_bootstrap.cc:654] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:28.090466 9252 log.cc:824] T ac7997f0dca54703af4ae3043094a888 P 351df5aa96a340afa40d099de3983611: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:28.090488 9250 log.cc:824] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:28.090490 9251 log.cc:824] T ac7997f0dca54703af4ae3043094a888 P f59ef63dbf234699bb291e0b2c1db841: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:28.091181 9250 tablet_bootstrap.cc:492] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401: No bootstrap required, opened a new log
I20260928 08:57:28.091254 9250 ts_tablet_manager.cc:1397] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:57:28.091323 9252 tablet_bootstrap.cc:492] T ac7997f0dca54703af4ae3043094a888 P 351df5aa96a340afa40d099de3983611: No bootstrap required, opened a new log
I20260928 08:57:28.091385 9252 ts_tablet_manager.cc:1397] T ac7997f0dca54703af4ae3043094a888 P 351df5aa96a340afa40d099de3983611: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:57:28.093357 9251 tablet_bootstrap.cc:492] T ac7997f0dca54703af4ae3043094a888 P f59ef63dbf234699bb291e0b2c1db841: No bootstrap required, opened a new log
I20260928 08:57:28.093240 9250 raft_consensus.cc:348] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75873050a9a6454b870709a7a20a8401" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 33385 } } peers { permanent_uuid: "f59ef63dbf234699bb291e0b2c1db841" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 37767 } } peers { permanent_uuid: "351df5aa96a340afa40d099de3983611" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34901 } }
I20260928 08:57:28.093288 9252 raft_consensus.cc:348] T ac7997f0dca54703af4ae3043094a888 P 351df5aa96a340afa40d099de3983611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75873050a9a6454b870709a7a20a8401" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 33385 } } peers { permanent_uuid: "f59ef63dbf234699bb291e0b2c1db841" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 37767 } } peers { permanent_uuid: "351df5aa96a340afa40d099de3983611" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34901 } }
I20260928 08:57:28.093418 9251 ts_tablet_manager.cc:1397] T ac7997f0dca54703af4ae3043094a888 P f59ef63dbf234699bb291e0b2c1db841: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 08:57:28.093425 9250 raft_consensus.cc:374] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:28.093425 9252 raft_consensus.cc:374] T ac7997f0dca54703af4ae3043094a888 P 351df5aa96a340afa40d099de3983611 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:28.093458 9252 raft_consensus.cc:729] T ac7997f0dca54703af4ae3043094a888 P 351df5aa96a340afa40d099de3983611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 351df5aa96a340afa40d099de3983611, State: Initialized, Role: FOLLOWER
I20260928 08:57:28.093458 9250 raft_consensus.cc:729] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75873050a9a6454b870709a7a20a8401, State: Initialized, Role: FOLLOWER
I20260928 08:57:28.093573 9252 consensus_queue.cc:260] T ac7997f0dca54703af4ae3043094a888 P 351df5aa96a340afa40d099de3983611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75873050a9a6454b870709a7a20a8401" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 33385 } } peers { permanent_uuid: "f59ef63dbf234699bb291e0b2c1db841" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 37767 } } peers { permanent_uuid: "351df5aa96a340afa40d099de3983611" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34901 } }
I20260928 08:57:28.093573 9250 consensus_queue.cc:260] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75873050a9a6454b870709a7a20a8401" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 33385 } } peers { permanent_uuid: "f59ef63dbf234699bb291e0b2c1db841" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 37767 } } peers { permanent_uuid: "351df5aa96a340afa40d099de3983611" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34901 } }
I20260928 08:57:28.093851 9250 ts_tablet_manager.cc:1428] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:57:28.093853 9102 heartbeater.cc:503] Master 127.23.167.254:41781 was elected leader, sending a full tablet report...
I20260928 08:57:28.093888 9252 ts_tablet_manager.cc:1428] T ac7997f0dca54703af4ae3043094a888 P 351df5aa96a340afa40d099de3983611: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 08:57:28.094003 9236 heartbeater.cc:503] Master 127.23.167.254:41781 was elected leader, sending a full tablet report...
I20260928 08:57:28.095257 9251 raft_consensus.cc:348] T ac7997f0dca54703af4ae3043094a888 P f59ef63dbf234699bb291e0b2c1db841 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75873050a9a6454b870709a7a20a8401" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 33385 } } peers { permanent_uuid: "f59ef63dbf234699bb291e0b2c1db841" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 37767 } } peers { permanent_uuid: "351df5aa96a340afa40d099de3983611" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34901 } }
I20260928 08:57:28.095386 9251 raft_consensus.cc:374] T ac7997f0dca54703af4ae3043094a888 P f59ef63dbf234699bb291e0b2c1db841 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:28.095417 9251 raft_consensus.cc:729] T ac7997f0dca54703af4ae3043094a888 P f59ef63dbf234699bb291e0b2c1db841 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f59ef63dbf234699bb291e0b2c1db841, State: Initialized, Role: FOLLOWER
I20260928 08:57:28.095518 9251 consensus_queue.cc:260] T ac7997f0dca54703af4ae3043094a888 P f59ef63dbf234699bb291e0b2c1db841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75873050a9a6454b870709a7a20a8401" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 33385 } } peers { permanent_uuid: "f59ef63dbf234699bb291e0b2c1db841" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 37767 } } peers { permanent_uuid: "351df5aa96a340afa40d099de3983611" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34901 } }
I20260928 08:57:28.095736 9251 ts_tablet_manager.cc:1428] T ac7997f0dca54703af4ae3043094a888 P f59ef63dbf234699bb291e0b2c1db841: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:57:28.095791 8968 heartbeater.cc:503] Master 127.23.167.254:41781 was elected leader, sending a full tablet report...
W20260928 08:57:28.160207 9103 tablet.cc:2369] T ac7997f0dca54703af4ae3043094a888 P 351df5aa96a340afa40d099de3983611: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:57:28.202625 9257 raft_consensus.cc:482] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:57:28.202757 9257 raft_consensus.cc:504] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75873050a9a6454b870709a7a20a8401" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 33385 } } peers { permanent_uuid: "f59ef63dbf234699bb291e0b2c1db841" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 37767 } } peers { permanent_uuid: "351df5aa96a340afa40d099de3983611" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34901 } }
I20260928 08:57:28.203224 9257 leader_election.cc:288] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f59ef63dbf234699bb291e0b2c1db841 (127.23.167.193:37767), 351df5aa96a340afa40d099de3983611 (127.23.167.194:34901)
I20260928 08:57:28.205991 9248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.203553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:50247 (local address 127.23.167.193:37767)
0928 08:57:28.203604 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:28.203606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:28.203614 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:57:28.203643 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:28.203644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:28.203656 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:28.203713 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:57:28.203869 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.204341 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.205279 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.205681 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.205749 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.205759 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:28.205888 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 08:57:28.205925 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 08:57:28.205991 9238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.203430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:37767 (local address 127.23.167.195:50247)
0928 08:57:28.203485 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:28.203499 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:28.203565 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:28.203738 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:28.203740 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:28.203742 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:57:28.203805 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:28.203807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.204421 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.204422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:28.205190 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:28.205192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.205749 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.205750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:28.205827 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.205830 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:57:28.205876 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 08:57:28.205917 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 08:57:28.206269 9249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.203591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:42447 (local address 127.23.167.194:34901)
0928 08:57:28.203652 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:28.203654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:28.203767 (+ 113us) server_negotiation.cc:407] Connection header received
0928 08:57:28.203812 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:28.203813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:28.203825 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:28.203891 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:57:28.204037 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.204533 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.205531 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.205908 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.205971 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.205979 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:28.206143 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 08:57:28.206203 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 08:57:28.206326 8922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7997f0dca54703af4ae3043094a888" candidate_uuid: "75873050a9a6454b870709a7a20a8401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f59ef63dbf234699bb291e0b2c1db841" is_pre_election: true
I20260928 08:57:28.206477 9259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.203521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:34901 (local address 127.23.167.195:42447)
0928 08:57:28.203703 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:28.203711 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:28.203763 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:28.203901 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:28.203901 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:28.203903 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:57:28.203976 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:28.203978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.204613 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.204614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:28.205447 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:28.205449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.205983 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.205984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:28.206053 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.206057 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:57:28.206360 (+ 303us) client_negotiation.cc:240] Negotiation successful
0928 08:57:28.206405 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":101,"threads_started":1}
I20260928 08:57:28.206506 8922 raft_consensus.cc:2454] T ac7997f0dca54703af4ae3043094a888 P f59ef63dbf234699bb291e0b2c1db841 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75873050a9a6454b870709a7a20a8401 in term 0.
I20260928 08:57:28.206763 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7997f0dca54703af4ae3043094a888" candidate_uuid: "75873050a9a6454b870709a7a20a8401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "351df5aa96a340afa40d099de3983611" is_pre_election: true
I20260928 08:57:28.206900 9056 raft_consensus.cc:2454] T ac7997f0dca54703af4ae3043094a888 P 351df5aa96a340afa40d099de3983611 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75873050a9a6454b870709a7a20a8401 in term 0.
I20260928 08:57:28.207062 9125 leader_election.cc:302] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75873050a9a6454b870709a7a20a8401, f59ef63dbf234699bb291e0b2c1db841; no voters:
I20260928 08:57:28.207168 9257 raft_consensus.cc:2781] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:57:28.207224 9257 raft_consensus.cc:482] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:57:28.207239 9257 raft_consensus.cc:3037] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:28.207953 9257 raft_consensus.cc:504] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75873050a9a6454b870709a7a20a8401" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 33385 } } peers { permanent_uuid: "f59ef63dbf234699bb291e0b2c1db841" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 37767 } } peers { permanent_uuid: "351df5aa96a340afa40d099de3983611" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34901 } }
I20260928 08:57:28.208065 9257 leader_election.cc:288] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401 [CANDIDATE]: Term 1 election: Requested vote from peers f59ef63dbf234699bb291e0b2c1db841 (127.23.167.193:37767), 351df5aa96a340afa40d099de3983611 (127.23.167.194:34901)
I20260928 08:57:28.208254 8922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7997f0dca54703af4ae3043094a888" candidate_uuid: "75873050a9a6454b870709a7a20a8401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f59ef63dbf234699bb291e0b2c1db841"
I20260928 08:57:28.208292 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7997f0dca54703af4ae3043094a888" candidate_uuid: "75873050a9a6454b870709a7a20a8401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "351df5aa96a340afa40d099de3983611"
I20260928 08:57:28.208323 8922 raft_consensus.cc:3037] T ac7997f0dca54703af4ae3043094a888 P f59ef63dbf234699bb291e0b2c1db841 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:28.208364 9056 raft_consensus.cc:3037] T ac7997f0dca54703af4ae3043094a888 P 351df5aa96a340afa40d099de3983611 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:28.209012 8922 raft_consensus.cc:2454] T ac7997f0dca54703af4ae3043094a888 P f59ef63dbf234699bb291e0b2c1db841 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75873050a9a6454b870709a7a20a8401 in term 1.
I20260928 08:57:28.209178 9056 raft_consensus.cc:2454] T ac7997f0dca54703af4ae3043094a888 P 351df5aa96a340afa40d099de3983611 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75873050a9a6454b870709a7a20a8401 in term 1.
I20260928 08:57:28.209184 9125 leader_election.cc:302] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75873050a9a6454b870709a7a20a8401, f59ef63dbf234699bb291e0b2c1db841; no voters:
I20260928 08:57:28.209275 9257 raft_consensus.cc:2781] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:57:28.209439 9257 raft_consensus.cc:686] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401 [term 1 LEADER]: Becoming Leader. State: Replica: 75873050a9a6454b870709a7a20a8401, State: Running, Role: LEADER
I20260928 08:57:28.209535 9257 consensus_queue.cc:237] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75873050a9a6454b870709a7a20a8401" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 33385 } } peers { permanent_uuid: "f59ef63dbf234699bb291e0b2c1db841" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 37767 } } peers { permanent_uuid: "351df5aa96a340afa40d099de3983611" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34901 } }
I20260928 08:57:28.210371 8777 catalog_manager.cc:5495] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75873050a9a6454b870709a7a20a8401 (127.23.167.195). New cstate: current_term: 1 leader_uuid: "75873050a9a6454b870709a7a20a8401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75873050a9a6454b870709a7a20a8401" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 33385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f59ef63dbf234699bb291e0b2c1db841" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 37767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "351df5aa96a340afa40d099de3983611" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34901 } health_report { overall_health: UNKNOWN } } }
I20260928 08:57:28.228431 9245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.226107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48854 (local address 127.23.167.195:33385)
0928 08:57:28.226150 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:28.226153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:28.226161 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:57:28.226237 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:28.226238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:28.226256 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:28.226313 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:57:28.226478 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.226956 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.227700 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.227890 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.227954 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.228092 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:57:28.228199 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:57:28.228272 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:28.228309 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 08:57:28.228331 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 08:57:28.230260 9056 raft_consensus.cc:1261] T ac7997f0dca54703af4ae3043094a888 P 351df5aa96a340afa40d099de3983611 [term 1 FOLLOWER]: Refusing update from remote peer 75873050a9a6454b870709a7a20a8401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:57:28.230284 8922 raft_consensus.cc:1261] T ac7997f0dca54703af4ae3043094a888 P f59ef63dbf234699bb291e0b2c1db841 [term 1 FOLLOWER]: Refusing update from remote peer 75873050a9a6454b870709a7a20a8401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:57:28.230611 9260 consensus_queue.cc:1035] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f59ef63dbf234699bb291e0b2c1db841" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 37767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:57:28.230739 9257 consensus_queue.cc:1035] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "351df5aa96a340afa40d099de3983611" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 34901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:57:28.233563 9264 mvcc.cc:205] Tried to move back new op lower bound from 7334239634348716032 to 7334239634266914816. Current Snapshot: MvccSnapshot[applied={T|T < 7334239634348716032}]
I20260928 08:57:28.233830 9265 mvcc.cc:205] Tried to move back new op lower bound from 7334239634348716032 to 7334239634266914816. Current Snapshot: MvccSnapshot[applied={T|T < 7334239634348716032}]
I20260928 08:57:28.235306 8777 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:57:28.235397 8777 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:57:28.236155 8777 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf735217cf704392a205fda76e244d40: Sending DeleteTablet for 3 replicas of tablet ac7997f0dca54703af4ae3043094a888
I20260928 08:57:28.236518 9170 tablet_service.cc:1514] Processing DeleteTablet for tablet ac7997f0dca54703af4ae3043094a888 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:57:28 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:48842
I20260928 08:57:28.236537 8902 tablet_service.cc:1514] Processing DeleteTablet for tablet ac7997f0dca54703af4ae3043094a888 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:57:28 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:33310
I20260928 08:57:28.236754 9275 tablet_replica.cc:330] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401: stopping tablet replica
I20260928 08:57:28.236745 9036 tablet_service.cc:1514] Processing DeleteTablet for tablet ac7997f0dca54703af4ae3043094a888 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:57:28 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:50694
I20260928 08:57:28.236848 9275 raft_consensus.cc:2229] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:57:28.236934 9275 raft_consensus.cc:2258] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:57:28.236951 9276 tablet_replica.cc:330] T ac7997f0dca54703af4ae3043094a888 P 351df5aa96a340afa40d099de3983611: stopping tablet replica
I20260928 08:57:28.237039 9276 raft_consensus.cc:2229] T ac7997f0dca54703af4ae3043094a888 P 351df5aa96a340afa40d099de3983611 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:57:28.237105 9274 tablet_replica.cc:330] T ac7997f0dca54703af4ae3043094a888 P f59ef63dbf234699bb291e0b2c1db841: stopping tablet replica
I20260928 08:57:28.237155 9276 raft_consensus.cc:2258] T ac7997f0dca54703af4ae3043094a888 P 351df5aa96a340afa40d099de3983611 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:57:28.237490 9275 ts_tablet_manager.cc:1905] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:57:28.237053 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 8839
I20260928 08:57:28.237803 9276 ts_tablet_manager.cc:1905] T ac7997f0dca54703af4ae3043094a888 P 351df5aa96a340afa40d099de3983611: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:57:28.239295 9275 ts_tablet_manager.cc:1918] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:57:28.239355 9276 ts_tablet_manager.cc:1918] T ac7997f0dca54703af4ae3043094a888 P 351df5aa96a340afa40d099de3983611: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:57:28.239372 9275 log.cc:1196] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-2/wal/wals/ac7997f0dca54703af4ae3043094a888
I20260928 08:57:28.239418 9276 log.cc:1196] T ac7997f0dca54703af4ae3043094a888 P 351df5aa96a340afa40d099de3983611: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585589999956-24223-0/minicluster-data/ts-1/wal/wals/ac7997f0dca54703af4ae3043094a888
I20260928 08:57:28.239650 9275 ts_tablet_manager.cc:1939] T ac7997f0dca54703af4ae3043094a888 P 75873050a9a6454b870709a7a20a8401: Deleting consensus metadata
I20260928 08:57:28.239673 9276 ts_tablet_manager.cc:1939] T ac7997f0dca54703af4ae3043094a888 P 351df5aa96a340afa40d099de3983611: Deleting consensus metadata
I20260928 08:57:28.240032 8763 catalog_manager.cc:4841] TS 75873050a9a6454b870709a7a20a8401 (127.23.167.195:33385): tablet ac7997f0dca54703af4ae3043094a888 (table test-table [id=d79ac7d7216c40ab9d84517c696ebadf]) successfully deleted
I20260928 08:57:28.240302 8763 catalog_manager.cc:4841] TS 351df5aa96a340afa40d099de3983611 (127.23.167.194:34901): tablet ac7997f0dca54703af4ae3043094a888 (table test-table [id=d79ac7d7216c40ab9d84517c696ebadf]) successfully deleted
W20260928 08:57:28.245999 8763 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.167.193:37767 (error 108)
I20260928 08:57:28.246523 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 8973
I20260928 08:57:28.246536 9246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.246399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:37767 (local address 127.0.0.1:33320)
0928 08:57:28.246451 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:28.246493 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.193:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 08:57:28.248804 8763 catalog_manager.cc:4584] TS f59ef63dbf234699bb291e0b2c1db841 (127.23.167.193:37767): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ac7997f0dca54703af4ae3043094a888: Network error: Client connection negotiation failed: client connection to 127.23.167.193:37767: connect: Connection refused (error 111)
I20260928 08:57:28.251870 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 9107
I20260928 08:57:28.256153 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 8745
2026-09-28T08:57:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1868 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T08:57:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:57:28Z Disabled control of system clock
I20260928 08:57:28.282063 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:43763
--webserver_interface=127.23.167.254
--webserver_port=0
--builtin_ntp_servers=127.23.167.212:42909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:43763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 08:57:28.362236 9281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:28.362462 9281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:28.362483 9281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:28.363636 9281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:57:28.363674 9281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:28.363685 9281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:28.363696 9281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:57:28.363708 9281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:57:28.365010 9281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:42909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:43763
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:43763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:28.365245 9281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:28.365530 9281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:28.370267 9289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:28.370424 9281 server_base.cc:1034] running on GCE node
W20260928 08:57:28.370263 9286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:28.370293 9287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:28.370800 9281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:28.371057 9281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:28.372225 9281 hybrid_clock.cc:648] HybridClock initialized: now 1790585848372208 us; error 28 us; skew 500 ppm
I20260928 08:57:28.373557 9281 webserver.cc:460] Webserver started at http://127.23.167.254:41715/ using document root <none> and password file <none>
I20260928 08:57:28.373778 9281 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:28.373827 9281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:28.373950 9281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:28.374976 9281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "57efd7a18c744910ac925c56b36e90c5"
format_stamp: "Formatted at 2026-09-28 08:57:28 on dist-test-slave-pbjp"
server_key: "ac7a35e348955a404cc2eb3cde306106"
server_key_iv: "5e66d723443138404b2ec3b6fd6b1caf"
server_key_version: "encryptionkey@0"
I20260928 08:57:28.375301 9281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "57efd7a18c744910ac925c56b36e90c5"
format_stamp: "Formatted at 2026-09-28 08:57:28 on dist-test-slave-pbjp"
server_key: "ac7a35e348955a404cc2eb3cde306106"
server_key_iv: "5e66d723443138404b2ec3b6fd6b1caf"
server_key_version: "encryptionkey@0"
I20260928 08:57:28.376510 9281 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:57:28.377154 9295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:28.377338 9281 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:28.377475 9281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "57efd7a18c744910ac925c56b36e90c5"
format_stamp: "Formatted at 2026-09-28 08:57:28 on dist-test-slave-pbjp"
server_key: "ac7a35e348955a404cc2eb3cde306106"
server_key_iv: "5e66d723443138404b2ec3b6fd6b1caf"
server_key_version: "encryptionkey@0"
I20260928 08:57:28.377550 9281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:28.402209 9281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:28.402550 9281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:28.402686 9281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:28.406594 9281 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:43763
I20260928 08:57:28.406682 9347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:43763 every 8 connection(s)
I20260928 08:57:28.406997 9281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:57:28.407915 9348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:28.410105 9348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57efd7a18c744910ac925c56b36e90c5: Bootstrap starting.
I20260928 08:57:28.410672 9348 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57efd7a18c744910ac925c56b36e90c5: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:28.410900 9348 log.cc:824] T 00000000000000000000000000000000 P 57efd7a18c744910ac925c56b36e90c5: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:28.411474 9348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57efd7a18c744910ac925c56b36e90c5: No bootstrap required, opened a new log
I20260928 08:57:28.413043 9348 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57efd7a18c744910ac925c56b36e90c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57efd7a18c744910ac925c56b36e90c5" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43763 } }
I20260928 08:57:28.413237 9348 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57efd7a18c744910ac925c56b36e90c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:28.413261 9348 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57efd7a18c744910ac925c56b36e90c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57efd7a18c744910ac925c56b36e90c5, State: Initialized, Role: FOLLOWER
I20260928 08:57:28.413378 9348 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57efd7a18c744910ac925c56b36e90c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57efd7a18c744910ac925c56b36e90c5" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43763 } }
I20260928 08:57:28.413450 9348 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57efd7a18c744910ac925c56b36e90c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:57:28.413477 9348 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57efd7a18c744910ac925c56b36e90c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:57:28.413511 9348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57efd7a18c744910ac925c56b36e90c5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:28.413949 9348 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57efd7a18c744910ac925c56b36e90c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57efd7a18c744910ac925c56b36e90c5" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43763 } }
I20260928 08:57:28.414036 9348 leader_election.cc:302] T 00000000000000000000000000000000 P 57efd7a18c744910ac925c56b36e90c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57efd7a18c744910ac925c56b36e90c5; no voters:
I20260928 08:57:28.414151 9348 leader_election.cc:288] T 00000000000000000000000000000000 P 57efd7a18c744910ac925c56b36e90c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:57:28.414201 9351 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57efd7a18c744910ac925c56b36e90c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:57:28.414354 9348 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57efd7a18c744910ac925c56b36e90c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:57:28.414378 9351 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57efd7a18c744910ac925c56b36e90c5 [term 1 LEADER]: Becoming Leader. State: Replica: 57efd7a18c744910ac925c56b36e90c5, State: Running, Role: LEADER
I20260928 08:57:28.414474 9351 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57efd7a18c744910ac925c56b36e90c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57efd7a18c744910ac925c56b36e90c5" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43763 } }
I20260928 08:57:28.414858 9352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57efd7a18c744910ac925c56b36e90c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57efd7a18c744910ac925c56b36e90c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57efd7a18c744910ac925c56b36e90c5" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43763 } } }
I20260928 08:57:28.414937 9353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57efd7a18c744910ac925c56b36e90c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57efd7a18c744910ac925c56b36e90c5. Latest consensus state: current_term: 1 leader_uuid: "57efd7a18c744910ac925c56b36e90c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57efd7a18c744910ac925c56b36e90c5" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43763 } } }
I20260928 08:57:28.414958 9352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57efd7a18c744910ac925c56b36e90c5 [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:28.414999 9353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57efd7a18c744910ac925c56b36e90c5 [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:28.415153 9360 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:57:28.415630 9360 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:57:28.416023 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 9281
I20260928 08:57:28.416116 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:57:28.416316 24223 external_mini_cluster.cc:1428] Setting key 86501fc962bf706a66e8c116f41a4b2c
I20260928 08:57:28.417083 9360 catalog_manager.cc:1347] Generated new cluster ID: 700da9a697944d7bb80527241bf294c2
I20260928 08:57:28.417133 9360 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:57:28.422709 9371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.416787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48390 (local address 127.23.167.254:43763)
0928 08:57:28.417376 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:28.417385 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:28.417413 (+ 28us) server_negotiation.cc:407] Connection header received
0928 08:57:28.417521 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:28.417528 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:28.417596 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:28.417723 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:28.418059 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.419441 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.420574 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.421221 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.421319 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.422037 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:28.422041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:28.422041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:28.422247 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:28.422287 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:28.422309 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:28.422365 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:28.422446 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 08:57:28.422519 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":73,"threads_started":1}
I20260928 08:57:28.442107 9360 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:57:28.442695 9360 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:57:28.457561 9360 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57efd7a18c744910ac925c56b36e90c5: Generated new TSK 0
I20260928 08:57:28.457762 9360 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:57:28.462412 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:0
--local_ip_for_outbound_sockets=127.23.167.193
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:43763
--builtin_ntp_servers=127.23.167.212:42909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 08:57:28.540174 9372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:28.540362 9372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:28.540381 9372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:28.541550 9372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:28.541589 9372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:28.541611 9372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:57:28.542906 9372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:42909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:43763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:28.543119 9372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:28.543385 9372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:28.548079 9377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:28.548141 9378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:28.548103 9380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:28.548252 9372 server_base.cc:1034] running on GCE node
I20260928 08:57:28.548488 9372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:28.548712 9372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:28.549844 9372 hybrid_clock.cc:648] HybridClock initialized: now 1790585848549830 us; error 33 us; skew 500 ppm
I20260928 08:57:28.551047 9372 webserver.cc:460] Webserver started at http://127.23.167.193:33879/ using document root <none> and password file <none>
I20260928 08:57:28.551257 9372 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:28.551306 9372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:28.551432 9372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:28.552503 9372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-0/data/instance:
uuid: "e0171b63f89a4004a94cfecc1f29b09b"
format_stamp: "Formatted at 2026-09-28 08:57:28 on dist-test-slave-pbjp"
server_key: "4282d5cf1ce6c1d7c6d60b4368d1b91e"
server_key_iv: "fa35f81e2349e9f1b9e112c64593605f"
server_key_version: "encryptionkey@0"
I20260928 08:57:28.552815 9372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance:
uuid: "e0171b63f89a4004a94cfecc1f29b09b"
format_stamp: "Formatted at 2026-09-28 08:57:28 on dist-test-slave-pbjp"
server_key: "4282d5cf1ce6c1d7c6d60b4368d1b91e"
server_key_iv: "fa35f81e2349e9f1b9e112c64593605f"
server_key_version: "encryptionkey@0"
I20260928 08:57:28.554020 9372 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:57:28.554695 9386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:28.554845 9372 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:57:28.554924 9372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "e0171b63f89a4004a94cfecc1f29b09b"
format_stamp: "Formatted at 2026-09-28 08:57:28 on dist-test-slave-pbjp"
server_key: "4282d5cf1ce6c1d7c6d60b4368d1b91e"
server_key_iv: "fa35f81e2349e9f1b9e112c64593605f"
server_key_version: "encryptionkey@0"
I20260928 08:57:28.554991 9372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:28.567142 9372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:28.567430 9372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:28.567559 9372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:28.567775 9372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:57:28.568076 9372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:57:28.568109 9372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:28.568137 9372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:57:28.568164 9372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:28.573513 9372 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:36233
I20260928 08:57:28.573575 9499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:36233 every 8 connection(s)
I20260928 08:57:28.573899 9372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
I20260928 08:57:28.576997 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 9372
I20260928 08:57:28.577111 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance
I20260928 08:57:28.577323 24223 external_mini_cluster.cc:1428] Setting key 68a8ffe536ccebfdecfc216942fb9334
I20260928 08:57:28.579130 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:0
--local_ip_for_outbound_sockets=127.23.167.194
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:43763
--builtin_ntp_servers=127.23.167.212:42909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 08:57:28.580487 9371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.574618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:41127 (local address 127.23.167.254:43763)
0928 08:57:28.574666 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:28.574667 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:28.574915 (+ 248us) server_negotiation.cc:407] Connection header received
0928 08:57:28.575593 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:28.575595 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:28.575613 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:28.575680 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:28.576555 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.577304 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.578833 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.579117 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.579192 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.580188 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:28.580189 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:28.580190 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:28.580195 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:28.580215 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:28.580220 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:28.580276 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:28.580366 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 08:57:28.580400 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 08:57:28.580721 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.574575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:43763 (local address 127.23.167.193:41127)
0928 08:57:28.574747 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:28.574802 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:28.575462 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:28.575779 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:28.575783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:28.575794 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:57:28.576454 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:28.576461 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.577498 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.577500 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:28.578737 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:28.578740 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.579201 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.579202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:28.579393 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.580047 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:57:28.580048 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:57:28.580078 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:57:28.580105 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:57:28.580284 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:57:28.580288 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:57:28.580292 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:57:28.580364 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 08:57:28.580491 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":64,"threads_started":1}
I20260928 08:57:28.581235 9500 heartbeater.cc:347] Connected to a master server at 127.23.167.254:43763
I20260928 08:57:28.581324 9500 heartbeater.cc:464] Registering TS with master...
I20260928 08:57:28.581521 9500 heartbeater.cc:511] Master 127.23.167.254:43763 requested a full tablet report, sending...
I20260928 08:57:28.582319 9312 ts_manager.cc:194] Registered new tserver with Master: e0171b63f89a4004a94cfecc1f29b09b (127.23.167.193:36233)
I20260928 08:57:28.583266 9312 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.167.193:41127
W20260928 08:57:28.660123 9503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:28.660355 9503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:28.660377 9503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:28.661634 9503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:28.661685 9503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:28.661712 9503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:57:28.663086 9503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:42909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:43763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:28.663293 9503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:28.663569 9503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:28.668411 9509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:28.668454 9511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:28.668625 9503 server_base.cc:1034] running on GCE node
W20260928 08:57:28.668490 9508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:28.668931 9503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:28.669198 9503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:28.670337 9503 hybrid_clock.cc:648] HybridClock initialized: now 1790585848670322 us; error 31 us; skew 500 ppm
I20260928 08:57:28.671614 9503 webserver.cc:460] Webserver started at http://127.23.167.194:33693/ using document root <none> and password file <none>
I20260928 08:57:28.671850 9503 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:28.671900 9503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:28.672041 9503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:28.673192 9503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-1/data/instance:
uuid: "15be76827d5a490281a38f641b597ff5"
format_stamp: "Formatted at 2026-09-28 08:57:28 on dist-test-slave-pbjp"
server_key: "c4743006fdb5bd7bc3f60a869a397f7c"
server_key_iv: "e818ee2e54b6cc2f60715c43dce37246"
server_key_version: "encryptionkey@0"
I20260928 08:57:28.673520 9503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance:
uuid: "15be76827d5a490281a38f641b597ff5"
format_stamp: "Formatted at 2026-09-28 08:57:28 on dist-test-slave-pbjp"
server_key: "c4743006fdb5bd7bc3f60a869a397f7c"
server_key_iv: "e818ee2e54b6cc2f60715c43dce37246"
server_key_version: "encryptionkey@0"
I20260928 08:57:28.674700 9503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:57:28.675364 9517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:28.675531 9503 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:57:28.675611 9503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "15be76827d5a490281a38f641b597ff5"
format_stamp: "Formatted at 2026-09-28 08:57:28 on dist-test-slave-pbjp"
server_key: "c4743006fdb5bd7bc3f60a869a397f7c"
server_key_iv: "e818ee2e54b6cc2f60715c43dce37246"
server_key_version: "encryptionkey@0"
I20260928 08:57:28.675671 9503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:28.698357 9503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:28.698649 9503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:28.698778 9503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:28.698982 9503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:57:28.699304 9503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:57:28.699352 9503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:28.699381 9503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:57:28.699404 9503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:28.704674 9503 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:45773
I20260928 08:57:28.704725 9630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:45773 every 8 connection(s)
I20260928 08:57:28.705039 9503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
I20260928 08:57:28.709831 9371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.705732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:55841 (local address 127.23.167.254:43763)
0928 08:57:28.705795 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:28.705797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:28.706027 (+ 230us) server_negotiation.cc:407] Connection header received
0928 08:57:28.706538 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:28.706539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:28.706549 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:28.706596 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:28.707203 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.707686 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.708630 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.708804 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.708847 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.709637 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:28.709638 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:28.709638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:28.709642 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:28.709656 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:28.709660 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:28.709701 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:28.709750 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 08:57:28.709773 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 08:57:28.710031 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.705721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:43763 (local address 127.23.167.194:55841)
0928 08:57:28.705908 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:28.705942 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:28.706457 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:28.706660 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:28.706664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:28.706674 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:57:28.707145 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:28.707148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.707752 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.707752 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:28.708560 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:28.708562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.708863 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.708864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:28.708971 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.709496 (+ 525us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:57:28.709496 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:57:28.709526 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:57:28.709550 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:57:28.709703 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:57:28.709706 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:57:28.709708 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 08:57:28.709752 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 08:57:28.709829 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20260928 08:57:28.710466 9631 heartbeater.cc:347] Connected to a master server at 127.23.167.254:43763
I20260928 08:57:28.710546 9631 heartbeater.cc:464] Registering TS with master...
I20260928 08:57:28.710691 9631 heartbeater.cc:511] Master 127.23.167.254:43763 requested a full tablet report, sending...
I20260928 08:57:28.711081 9312 ts_manager.cc:194] Registered new tserver with Master: 15be76827d5a490281a38f641b597ff5 (127.23.167.194:45773)
I20260928 08:57:28.711514 9312 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.167.194:55841
I20260928 08:57:28.714891 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 9503
I20260928 08:57:28.714980 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance
I20260928 08:57:28.715166 24223 external_mini_cluster.cc:1428] Setting key ee5e1a2cd79f9751e9dc20acb0135556
I20260928 08:57:28.716048 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:0
--local_ip_for_outbound_sockets=127.23.167.195
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:43763
--builtin_ntp_servers=127.23.167.212:42909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 08:57:28.795450 9634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:28.795645 9634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:28.795665 9634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:28.796741 9634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:28.796772 9634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:28.796794 9634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:57:28.798054 9634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:42909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:43763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:28.798238 9634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:28.798482 9634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:28.803206 9639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:28.803228 9642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:28.803213 9640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:28.803371 9634 server_base.cc:1034] running on GCE node
I20260928 08:57:28.803609 9634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:28.803799 9634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:28.804960 9634 hybrid_clock.cc:648] HybridClock initialized: now 1790585848804942 us; error 35 us; skew 500 ppm
I20260928 08:57:28.806123 9634 webserver.cc:460] Webserver started at http://127.23.167.195:45095/ using document root <none> and password file <none>
I20260928 08:57:28.806303 9634 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:28.806340 9634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:28.806430 9634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:28.807425 9634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-2/data/instance:
uuid: "22610c0698514305aa69f87e00a6f38f"
format_stamp: "Formatted at 2026-09-28 08:57:28 on dist-test-slave-pbjp"
server_key: "0e12e262d3411434f2e903fe1dd9ebb1"
server_key_iv: "12ea1f5e8c63bd2d72e1e1ad11efa219"
server_key_version: "encryptionkey@0"
I20260928 08:57:28.807722 9634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance:
uuid: "22610c0698514305aa69f87e00a6f38f"
format_stamp: "Formatted at 2026-09-28 08:57:28 on dist-test-slave-pbjp"
server_key: "0e12e262d3411434f2e903fe1dd9ebb1"
server_key_iv: "12ea1f5e8c63bd2d72e1e1ad11efa219"
server_key_version: "encryptionkey@0"
I20260928 08:57:28.808880 9634 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:57:28.809607 9648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:28.809792 9634 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 08:57:28.809873 9634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "22610c0698514305aa69f87e00a6f38f"
format_stamp: "Formatted at 2026-09-28 08:57:28 on dist-test-slave-pbjp"
server_key: "0e12e262d3411434f2e903fe1dd9ebb1"
server_key_iv: "12ea1f5e8c63bd2d72e1e1ad11efa219"
server_key_version: "encryptionkey@0"
I20260928 08:57:28.809938 9634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:28.839398 9634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:28.839704 9634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:28.839810 9634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:28.840015 9634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:57:28.840286 9634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:57:28.840322 9634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:28.840348 9634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:57:28.840364 9634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:28.846020 9634 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:41735
I20260928 08:57:28.846093 9761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:41735 every 8 connection(s)
I20260928 08:57:28.846377 9634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
I20260928 08:57:28.850793 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 9634
I20260928 08:57:28.850901 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance
I20260928 08:57:28.851092 24223 external_mini_cluster.cc:1428] Setting key 2438c848f96b3e1ed8c329d437f3c19b
I20260928 08:57:28.851583 9371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.847096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:52245 (local address 127.23.167.254:43763)
0928 08:57:28.847143 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:28.847145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:28.847411 (+ 266us) server_negotiation.cc:407] Connection header received
0928 08:57:28.847956 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:28.847958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:28.847972 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:28.848026 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:28.848715 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.849231 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.850203 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.850399 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.850454 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.851320 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:28.851321 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:28.851321 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:28.851333 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:28.851348 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:28.851352 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:28.851393 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:28.851471 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 08:57:28.851505 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 08:57:28.851788 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.847054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:43763 (local address 127.23.167.195:52245)
0928 08:57:28.847280 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:28.847316 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:28.847853 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:28.848112 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:28.848116 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:28.848124 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:57:28.848622 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:28.848627 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.849314 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.849315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:28.850119 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:28.850121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.850469 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.850470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:28.850619 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.851166 (+ 547us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:57:28.851167 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:57:28.851206 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:57:28.851236 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:57:28.851399 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:57:28.851404 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:57:28.851407 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:57:28.851470 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 08:57:28.851576 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":96,"threads_started":1}
I20260928 08:57:28.852190 9762 heartbeater.cc:347] Connected to a master server at 127.23.167.254:43763
I20260928 08:57:28.852270 9762 heartbeater.cc:464] Registering TS with master...
I20260928 08:57:28.852442 9762 heartbeater.cc:511] Master 127.23.167.254:43763 requested a full tablet report, sending...
I20260928 08:57:28.852797 9312 ts_manager.cc:194] Registered new tserver with Master: 22610c0698514305aa69f87e00a6f38f (127.23.167.195:41735)
I20260928 08:57:28.853186 9312 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.167.195:52245
I20260928 08:57:28.862979 24223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:57:28.866942 9371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.864326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48404 (local address 127.23.167.254:43763)
0928 08:57:28.864404 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:28.864406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:28.864451 (+ 45us) server_negotiation.cc:407] Connection header received
0928 08:57:28.864507 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:28.864507 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:28.864518 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:28.864563 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:28.864694 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.865227 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.865960 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.866156 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.866210 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.866739 (+ 529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:28.866740 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:28.866741 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:28.866745 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:28.866758 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:28.866763 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:28.866801 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:28.866840 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 08:57:28.866863 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 08:57:28.868288 9312 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:57:28.869915 9312 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:57:28.875909 9775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.872317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51378 (local address 127.23.167.195:41735)
0928 08:57:28.872532 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:28.872535 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:28.872676 (+ 141us) server_negotiation.cc:407] Connection header received
0928 08:57:28.872733 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:28.872735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:28.872749 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:28.872815 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:57:28.872974 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.874053 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.875057 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.875538 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.875601 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.875693 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:28.875782 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 08:57:28.875823 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":134,"threads_started":1}
I20260928 08:57:28.875902 9774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.872126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:41735 (local address 127.0.0.1:51378)
0928 08:57:28.872603 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:28.872612 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:28.872669 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:28.872875 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:28.872875 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:28.872877 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:57:28.872908 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:28.872910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.874130 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.874131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:28.874980 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:28.874982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.875611 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.875612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:28.875686 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.875699 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 08:57:28.875768 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 08:57:28.875820 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":63,"threads_started":1}
I20260928 08:57:28.876000 9770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.871922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:45773 (local address 127.0.0.1:59980)
0928 08:57:28.872240 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:28.872269 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:28.872331 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:28.872553 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:28.872556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:28.872558 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:57:28.872624 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:28.872625 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.873640 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.873641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:28.874751 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:28.874754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.875752 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.875753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:28.875814 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.875818 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:57:28.875878 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 08:57:28.875920 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
I20260928 08:57:28.876289 9772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.872005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59980 (local address 127.23.167.194:45773)
0928 08:57:28.872202 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:28.872208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:28.872405 (+ 197us) server_negotiation.cc:407] Connection header received
0928 08:57:28.872450 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:28.872453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:28.872469 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:28.872536 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:57:28.872698 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.873565 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.874856 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.875575 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.875639 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.875718 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:28.876163 (+ 445us) server_negotiation.cc:299] Negotiation successful
0928 08:57:28.876206 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":99,"threads_started":1}
I20260928 08:57:28.876348 9771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.871922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:36233 (local address 127.0.0.1:59100)
0928 08:57:28.872215 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:28.872259 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:28.872329 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:28.872730 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:28.872731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:28.872732 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:57:28.872779 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:28.872781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.874110 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.874111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:28.875204 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:28.875207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.876145 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.876147 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:28.876204 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.876207 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:57:28.876243 (+ 36us) client_negotiation.cc:240] Negotiation successful
0928 08:57:28.876278 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":60,"threads_started":1}
I20260928 08:57:28.876576 9773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.872091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59100 (local address 127.23.167.193:36233)
0928 08:57:28.872360 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:28.872365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:28.872387 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:57:28.872424 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:28.872426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:28.872440 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:28.872516 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:57:28.872852 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.874023 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.875287 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.875987 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.876045 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.876456 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:28.876478 (+ 22us) server_negotiation.cc:299] Negotiation successful
0928 08:57:28.876507 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":70,"threads_started":1}
I20260928 08:57:28.876740 9696 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7ae4fd1b9249a7b0f0fdcb7c628d04 (DEFAULT_TABLE table=test-table [id=fa5fdc9afb434aa0a96c973e8d49e16d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:57:28.877158 9696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c7ae4fd1b9249a7b0f0fdcb7c628d04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:28.878938 9565 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7ae4fd1b9249a7b0f0fdcb7c628d04 (DEFAULT_TABLE table=test-table [id=fa5fdc9afb434aa0a96c973e8d49e16d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:57:28.879182 9434 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7ae4fd1b9249a7b0f0fdcb7c628d04 (DEFAULT_TABLE table=test-table [id=fa5fdc9afb434aa0a96c973e8d49e16d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:57:28.879317 9565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c7ae4fd1b9249a7b0f0fdcb7c628d04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:28.879571 9434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c7ae4fd1b9249a7b0f0fdcb7c628d04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:28.880106 9776 tablet_bootstrap.cc:492] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f: Bootstrap starting.
I20260928 08:57:28.880707 9776 tablet_bootstrap.cc:654] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:28.881028 9776 log.cc:824] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:28.881801 9776 tablet_bootstrap.cc:492] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f: No bootstrap required, opened a new log
I20260928 08:57:28.881860 9776 ts_tablet_manager.cc:1397] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:57:28.882663 9778 tablet_bootstrap.cc:492] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 15be76827d5a490281a38f641b597ff5: Bootstrap starting.
I20260928 08:57:28.882887 9779 tablet_bootstrap.cc:492] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P e0171b63f89a4004a94cfecc1f29b09b: Bootstrap starting.
I20260928 08:57:28.883296 9778 tablet_bootstrap.cc:654] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 15be76827d5a490281a38f641b597ff5: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:28.883437 9779 tablet_bootstrap.cc:654] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P e0171b63f89a4004a94cfecc1f29b09b: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:28.883596 9778 log.cc:824] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 15be76827d5a490281a38f641b597ff5: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:28.883723 9779 log.cc:824] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P e0171b63f89a4004a94cfecc1f29b09b: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:28.884191 9776 raft_consensus.cc:348] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0171b63f89a4004a94cfecc1f29b09b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 36233 } } peers { permanent_uuid: "15be76827d5a490281a38f641b597ff5" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 45773 } } peers { permanent_uuid: "22610c0698514305aa69f87e00a6f38f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 41735 } }
I20260928 08:57:28.884315 9778 tablet_bootstrap.cc:492] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 15be76827d5a490281a38f641b597ff5: No bootstrap required, opened a new log
I20260928 08:57:28.884356 9776 raft_consensus.cc:374] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:28.884392 9776 raft_consensus.cc:729] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22610c0698514305aa69f87e00a6f38f, State: Initialized, Role: FOLLOWER
I20260928 08:57:28.884395 9778 ts_tablet_manager.cc:1397] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 15be76827d5a490281a38f641b597ff5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:57:28.884445 9779 tablet_bootstrap.cc:492] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P e0171b63f89a4004a94cfecc1f29b09b: No bootstrap required, opened a new log
I20260928 08:57:28.884497 9779 ts_tablet_manager.cc:1397] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P e0171b63f89a4004a94cfecc1f29b09b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:57:28.884543 9776 consensus_queue.cc:260] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0171b63f89a4004a94cfecc1f29b09b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 36233 } } peers { permanent_uuid: "15be76827d5a490281a38f641b597ff5" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 45773 } } peers { permanent_uuid: "22610c0698514305aa69f87e00a6f38f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 41735 } }
I20260928 08:57:28.884792 9776 ts_tablet_manager.cc:1428] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:57:28.884991 9762 heartbeater.cc:503] Master 127.23.167.254:43763 was elected leader, sending a full tablet report...
I20260928 08:57:28.886490 9778 raft_consensus.cc:348] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 15be76827d5a490281a38f641b597ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0171b63f89a4004a94cfecc1f29b09b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 36233 } } peers { permanent_uuid: "15be76827d5a490281a38f641b597ff5" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 45773 } } peers { permanent_uuid: "22610c0698514305aa69f87e00a6f38f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 41735 } }
I20260928 08:57:28.886610 9778 raft_consensus.cc:374] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 15be76827d5a490281a38f641b597ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:28.886641 9778 raft_consensus.cc:729] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 15be76827d5a490281a38f641b597ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15be76827d5a490281a38f641b597ff5, State: Initialized, Role: FOLLOWER
I20260928 08:57:28.886747 9778 consensus_queue.cc:260] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 15be76827d5a490281a38f641b597ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0171b63f89a4004a94cfecc1f29b09b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 36233 } } peers { permanent_uuid: "15be76827d5a490281a38f641b597ff5" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 45773 } } peers { permanent_uuid: "22610c0698514305aa69f87e00a6f38f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 41735 } }
I20260928 08:57:28.886821 9779 raft_consensus.cc:348] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P e0171b63f89a4004a94cfecc1f29b09b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0171b63f89a4004a94cfecc1f29b09b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 36233 } } peers { permanent_uuid: "15be76827d5a490281a38f641b597ff5" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 45773 } } peers { permanent_uuid: "22610c0698514305aa69f87e00a6f38f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 41735 } }
I20260928 08:57:28.886965 9779 raft_consensus.cc:374] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P e0171b63f89a4004a94cfecc1f29b09b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:28.886993 9778 ts_tablet_manager.cc:1428] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 15be76827d5a490281a38f641b597ff5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:57:28.887010 9631 heartbeater.cc:503] Master 127.23.167.254:43763 was elected leader, sending a full tablet report...
I20260928 08:57:28.887012 9779 raft_consensus.cc:729] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P e0171b63f89a4004a94cfecc1f29b09b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0171b63f89a4004a94cfecc1f29b09b, State: Initialized, Role: FOLLOWER
I20260928 08:57:28.887115 9779 consensus_queue.cc:260] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P e0171b63f89a4004a94cfecc1f29b09b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0171b63f89a4004a94cfecc1f29b09b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 36233 } } peers { permanent_uuid: "15be76827d5a490281a38f641b597ff5" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 45773 } } peers { permanent_uuid: "22610c0698514305aa69f87e00a6f38f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 41735 } }
I20260928 08:57:28.887341 9779 ts_tablet_manager.cc:1428] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P e0171b63f89a4004a94cfecc1f29b09b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:57:28.887375 9500 heartbeater.cc:503] Master 127.23.167.254:43763 was elected leader, sending a full tablet report...
I20260928 08:57:28.917941 9782 raft_consensus.cc:482] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:57:28.918058 9782 raft_consensus.cc:504] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0171b63f89a4004a94cfecc1f29b09b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 36233 } } peers { permanent_uuid: "15be76827d5a490281a38f641b597ff5" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 45773 } } peers { permanent_uuid: "22610c0698514305aa69f87e00a6f38f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 41735 } }
I20260928 08:57:28.918432 9782 leader_election.cc:288] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0171b63f89a4004a94cfecc1f29b09b (127.23.167.193:36233), 15be76827d5a490281a38f641b597ff5 (127.23.167.194:45773)
I20260928 08:57:28.922209 9773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.918724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:47587 (local address 127.23.167.193:36233)
0928 08:57:28.918765 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:28.918767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:28.918775 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:57:28.918803 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:28.918803 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:28.918814 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:28.918883 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:57:28.919036 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.919795 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.921316 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.921881 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.921960 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.921968 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:28.922101 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 08:57:28.922139 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 08:57:28.922209 9772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.918672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:51015 (local address 127.23.167.194:45773)
0928 08:57:28.918798 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:28.918800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:28.918829 (+ 29us) server_negotiation.cc:407] Connection header received
0928 08:57:28.918874 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:28.918874 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:28.918886 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:28.918936 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:57:28.919050 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.919819 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.921322 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.921886 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.921961 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.921968 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:28.922101 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 08:57:28.922139 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260928 08:57:28.922231 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.918569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:36233 (local address 127.23.167.195:47587)
0928 08:57:28.918621 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:28.918633 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:28.918673 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:28.918894 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:28.918895 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:28.918897 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:57:28.918968 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:28.918970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.919893 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.919894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:28.921219 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:28.921222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.921961 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.921961 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:28.922039 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.922043 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:57:28.922087 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 08:57:28.922130 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":55}
I20260928 08:57:28.922241 9785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.918570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:45773 (local address 127.23.167.195:51015)
0928 08:57:28.918781 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:28.918789 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:28.918834 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:28.918935 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:28.918935 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:28.918937 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:57:28.918984 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:28.918986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.919893 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.919894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:28.921219 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:28.921223 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.921962 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.921962 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:28.922041 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.922043 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 08:57:28.922092 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 08:57:28.922129 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":31,"thread_start_us":92,"threads_started":1}
I20260928 08:57:28.922591 9454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7ae4fd1b9249a7b0f0fdcb7c628d04" candidate_uuid: "22610c0698514305aa69f87e00a6f38f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0171b63f89a4004a94cfecc1f29b09b" is_pre_election: true
I20260928 08:57:28.922777 9454 raft_consensus.cc:2454] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P e0171b63f89a4004a94cfecc1f29b09b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22610c0698514305aa69f87e00a6f38f in term 0.
I20260928 08:57:28.922740 9585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7ae4fd1b9249a7b0f0fdcb7c628d04" candidate_uuid: "22610c0698514305aa69f87e00a6f38f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15be76827d5a490281a38f641b597ff5" is_pre_election: true
I20260928 08:57:28.922932 9585 raft_consensus.cc:2454] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 15be76827d5a490281a38f641b597ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22610c0698514305aa69f87e00a6f38f in term 0.
I20260928 08:57:28.923178 9650 leader_election.cc:302] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15be76827d5a490281a38f641b597ff5, 22610c0698514305aa69f87e00a6f38f; no voters:
I20260928 08:57:28.923352 9782 raft_consensus.cc:2781] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:57:28.923408 9782 raft_consensus.cc:482] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:57:28.923424 9782 raft_consensus.cc:3037] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:28.924072 9782 raft_consensus.cc:504] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0171b63f89a4004a94cfecc1f29b09b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 36233 } } peers { permanent_uuid: "15be76827d5a490281a38f641b597ff5" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 45773 } } peers { permanent_uuid: "22610c0698514305aa69f87e00a6f38f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 41735 } }
I20260928 08:57:28.924180 9782 leader_election.cc:288] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f [CANDIDATE]: Term 1 election: Requested vote from peers e0171b63f89a4004a94cfecc1f29b09b (127.23.167.193:36233), 15be76827d5a490281a38f641b597ff5 (127.23.167.194:45773)
I20260928 08:57:28.924320 9454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7ae4fd1b9249a7b0f0fdcb7c628d04" candidate_uuid: "22610c0698514305aa69f87e00a6f38f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0171b63f89a4004a94cfecc1f29b09b"
I20260928 08:57:28.924340 9585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7ae4fd1b9249a7b0f0fdcb7c628d04" candidate_uuid: "22610c0698514305aa69f87e00a6f38f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15be76827d5a490281a38f641b597ff5"
I20260928 08:57:28.924402 9585 raft_consensus.cc:3037] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 15be76827d5a490281a38f641b597ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:28.924410 9454 raft_consensus.cc:3037] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P e0171b63f89a4004a94cfecc1f29b09b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:28.925303 9454 raft_consensus.cc:2454] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P e0171b63f89a4004a94cfecc1f29b09b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22610c0698514305aa69f87e00a6f38f in term 1.
I20260928 08:57:28.925303 9585 raft_consensus.cc:2454] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 15be76827d5a490281a38f641b597ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22610c0698514305aa69f87e00a6f38f in term 1.
I20260928 08:57:28.925499 9652 leader_election.cc:302] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22610c0698514305aa69f87e00a6f38f, e0171b63f89a4004a94cfecc1f29b09b; no voters:
I20260928 08:57:28.925598 9782 raft_consensus.cc:2781] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:57:28.925741 9782 raft_consensus.cc:686] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f [term 1 LEADER]: Becoming Leader. State: Replica: 22610c0698514305aa69f87e00a6f38f, State: Running, Role: LEADER
I20260928 08:57:28.925830 9782 consensus_queue.cc:237] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0171b63f89a4004a94cfecc1f29b09b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 36233 } } peers { permanent_uuid: "15be76827d5a490281a38f641b597ff5" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 45773 } } peers { permanent_uuid: "22610c0698514305aa69f87e00a6f38f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 41735 } }
I20260928 08:57:28.926652 9311 catalog_manager.cc:5495] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f reported cstate change: term changed from 0 to 1, leader changed from <none> to 22610c0698514305aa69f87e00a6f38f (127.23.167.195). New cstate: current_term: 1 leader_uuid: "22610c0698514305aa69f87e00a6f38f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0171b63f89a4004a94cfecc1f29b09b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 36233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15be76827d5a490281a38f641b597ff5" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 45773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22610c0698514305aa69f87e00a6f38f" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260928 08:57:28.934628 9775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.932472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51394 (local address 127.23.167.195:41735)
0928 08:57:28.932521 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:28.932522 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:28.932529 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:57:28.932603 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:28.932604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:28.932618 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:28.932676 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:57:28.932819 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.933312 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.934048 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:28.934229 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:28.934271 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:28.934360 (+ 89us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:57:28.934452 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:57:28.934504 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:28.934523 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 08:57:28.934536 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 08:57:28.936362 9585 raft_consensus.cc:1261] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 15be76827d5a490281a38f641b597ff5 [term 1 FOLLOWER]: Refusing update from remote peer 22610c0698514305aa69f87e00a6f38f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:57:28.936440 9454 raft_consensus.cc:1261] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P e0171b63f89a4004a94cfecc1f29b09b [term 1 FOLLOWER]: Refusing update from remote peer 22610c0698514305aa69f87e00a6f38f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:57:28.936713 9786 consensus_queue.cc:1035] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f [LEADER]: Connected to new peer: Peer: permanent_uuid: "15be76827d5a490281a38f641b597ff5" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 45773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:57:28.936833 9782 consensus_queue.cc:1035] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0171b63f89a4004a94cfecc1f29b09b" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 36233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:57:28.939679 9791 mvcc.cc:205] Tried to move back new op lower bound from 7334239637240696832 to 7334239637200883712. Current Snapshot: MvccSnapshot[applied={T|T < 7334239637240696832}]
I20260928 08:57:28.939952 9790 mvcc.cc:205] Tried to move back new op lower bound from 7334239637240696832 to 7334239637200883712. Current Snapshot: MvccSnapshot[applied={T|T < 7334239637240696832}]
I20260928 08:57:28.941896 9311 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:57:28.941985 9311 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:57:28.942759 9311 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57efd7a18c744910ac925c56b36e90c5: Sending DeleteTablet for 3 replicas of tablet 8c7ae4fd1b9249a7b0f0fdcb7c628d04
I20260928 08:57:28.943159 9696 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c7ae4fd1b9249a7b0f0fdcb7c628d04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:57:28 UTC) from {username='slave'} at 127.0.0.1:51378
I20260928 08:57:28.943190 9565 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c7ae4fd1b9249a7b0f0fdcb7c628d04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:57:28 UTC) from {username='slave'} at 127.0.0.1:59980
I20260928 08:57:28.943435 9799 tablet_replica.cc:330] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 15be76827d5a490281a38f641b597ff5: stopping tablet replica
I20260928 08:57:28.943535 9799 raft_consensus.cc:2229] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 15be76827d5a490281a38f641b597ff5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:57:28.943509 9434 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c7ae4fd1b9249a7b0f0fdcb7c628d04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:57:28 UTC) from {username='slave'} at 127.0.0.1:59100
I20260928 08:57:28.943634 9799 raft_consensus.cc:2258] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 15be76827d5a490281a38f641b597ff5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:57:28.943907 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 9372
I20260928 08:57:28.943971 9800 tablet_replica.cc:330] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P e0171b63f89a4004a94cfecc1f29b09b: stopping tablet replica
I20260928 08:57:28.944114 9800 raft_consensus.cc:2229] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P e0171b63f89a4004a94cfecc1f29b09b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:57:28.944253 9800 raft_consensus.cc:2258] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P e0171b63f89a4004a94cfecc1f29b09b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:57:28.944478 9798 tablet_replica.cc:330] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f: stopping tablet replica
I20260928 08:57:28.944639 9798 raft_consensus.cc:2229] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:57:28.944803 9798 raft_consensus.cc:2258] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:57:28.945289 9798 ts_tablet_manager.cc:1905] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:57:28.945302 9799 ts_tablet_manager.cc:1905] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 15be76827d5a490281a38f641b597ff5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:57:28.947170 9798 ts_tablet_manager.cc:1918] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:57:28.947245 9798 log.cc:1196] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-2/wal/wals/8c7ae4fd1b9249a7b0f0fdcb7c628d04
I20260928 08:57:28.947475 9798 ts_tablet_manager.cc:1939] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 22610c0698514305aa69f87e00a6f38f: Deleting consensus metadata
I20260928 08:57:28.947592 9799 ts_tablet_manager.cc:1918] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 15be76827d5a490281a38f641b597ff5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:57:28.947712 9799 log.cc:1196] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 15be76827d5a490281a38f641b597ff5: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585589999956-24223-0/minicluster-data/ts-1/wal/wals/8c7ae4fd1b9249a7b0f0fdcb7c628d04
I20260928 08:57:28.947868 9299 catalog_manager.cc:4841] TS 22610c0698514305aa69f87e00a6f38f (127.23.167.195:41735): tablet 8c7ae4fd1b9249a7b0f0fdcb7c628d04 (table test-table [id=fa5fdc9afb434aa0a96c973e8d49e16d]) successfully deleted
I20260928 08:57:28.948190 9799 ts_tablet_manager.cc:1939] T 8c7ae4fd1b9249a7b0f0fdcb7c628d04 P 15be76827d5a490281a38f641b597ff5: Deleting consensus metadata
I20260928 08:57:28.948930 9297 catalog_manager.cc:4841] TS 15be76827d5a490281a38f641b597ff5 (127.23.167.194:45773): tablet 8c7ae4fd1b9249a7b0f0fdcb7c628d04 (table test-table [id=fa5fdc9afb434aa0a96c973e8d49e16d]) successfully deleted
W20260928 08:57:28.951260 9299 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.167.193:36233 (error 108)
I20260928 08:57:28.951380 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 9503
I20260928 08:57:28.953730 9774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:28.953588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:36233 (local address 127.0.0.1:59106)
0928 08:57:28.953642 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:28.953688 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.193:36233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 08:57:28.953842 9299 catalog_manager.cc:4584] TS e0171b63f89a4004a94cfecc1f29b09b (127.23.167.193:36233): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8c7ae4fd1b9249a7b0f0fdcb7c628d04: Network error: Client connection negotiation failed: client connection to 127.23.167.193:36233: connect: Connection refused (error 111)
I20260928 08:57:28.956363 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 9634
I20260928 08:57:28.960237 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 9281
2026-09-28T08:57:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (702 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:57:29 dist-test-slave-pbjp krb5kdc[9804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:57:29 dist-test-slave-pbjp krb5kdc[9804](info): set up 2 sockets
Sep 28 08:57:29 dist-test-slave-pbjp krb5kdc[9804](info): commencing operation
krb5kdc: starting...
W20260928 08:57:29.916723 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:57:29 dist-test-slave-pbjp krb5kdc[9804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:57:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:57:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.167.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
I20260928 08:57:30.028337 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:38977
--webserver_interface=127.23.167.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:38329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:38977
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:30.106437 9817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:30.106652 9817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:30.106706 9817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:30.107864 9817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:57:30.107908 9817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:30.107925 9817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:30.107942 9817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:57:30.107959 9817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:57:30.109477 9817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:38329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:38977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:38977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:30.109736 9817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:30.109944 9817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:30.114758 9823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:30.114782 9825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:30.114986 9817 server_base.cc:1034] running on GCE node
W20260928 08:57:30.114785 9822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:30.115461 9817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:30.115752 9817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:30.116889 9817 hybrid_clock.cc:648] HybridClock initialized: now 1790585850116875 us; error 37 us; skew 500 ppm
Sep 28 08:57:30 dist-test-slave-pbjp krb5kdc[9804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585850, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:30.119345 9817 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:57:30.119792 9817 webserver.cc:460] Webserver started at http://127.23.167.254:43183/ using document root <none> and password file <none>
I20260928 08:57:30.120012 9817 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:30.120061 9817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:30.120184 9817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:30.121281 9817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "8d5789a8a6e7449d96a57c12166788bf"
format_stamp: "Formatted at 2026-09-28 08:57:30 on dist-test-slave-pbjp"
server_key: "cf2954402c331a94d9ab8b938051fd80"
server_key_iv: "63edb58af785fdb7605a9527ca1df2d6"
server_key_version: "encryptionkey@0"
I20260928 08:57:30.121615 9817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "8d5789a8a6e7449d96a57c12166788bf"
format_stamp: "Formatted at 2026-09-28 08:57:30 on dist-test-slave-pbjp"
server_key: "cf2954402c331a94d9ab8b938051fd80"
server_key_iv: "63edb58af785fdb7605a9527ca1df2d6"
server_key_version: "encryptionkey@0"
I20260928 08:57:30.122881 9817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:57:30.123618 9832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:30.123788 9817 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:57:30.123871 9817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "8d5789a8a6e7449d96a57c12166788bf"
format_stamp: "Formatted at 2026-09-28 08:57:30 on dist-test-slave-pbjp"
server_key: "cf2954402c331a94d9ab8b938051fd80"
server_key_iv: "63edb58af785fdb7605a9527ca1df2d6"
server_key_version: "encryptionkey@0"
I20260928 08:57:30.123929 9817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:30.147598 9817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:30.149653 9817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:30.149796 9817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:30.153563 9817 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:38977
I20260928 08:57:30.153628 9884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:38977 every 8 connection(s)
I20260928 08:57:30.153949 9817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:57:30.154832 9885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:30.157195 9885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d5789a8a6e7449d96a57c12166788bf: Bootstrap starting.
I20260928 08:57:30.157770 9885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d5789a8a6e7449d96a57c12166788bf: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:30.158042 9885 log.cc:824] T 00000000000000000000000000000000 P 8d5789a8a6e7449d96a57c12166788bf: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:30.158751 9885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d5789a8a6e7449d96a57c12166788bf: No bootstrap required, opened a new log
I20260928 08:57:30.160342 9885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d5789a8a6e7449d96a57c12166788bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d5789a8a6e7449d96a57c12166788bf" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38977 } }
I20260928 08:57:30.160501 9885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d5789a8a6e7449d96a57c12166788bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:30.160534 9885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d5789a8a6e7449d96a57c12166788bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d5789a8a6e7449d96a57c12166788bf, State: Initialized, Role: FOLLOWER
I20260928 08:57:30.160689 9885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d5789a8a6e7449d96a57c12166788bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d5789a8a6e7449d96a57c12166788bf" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38977 } }
I20260928 08:57:30.160769 9885 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d5789a8a6e7449d96a57c12166788bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:57:30.160802 9885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d5789a8a6e7449d96a57c12166788bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:57:30.160853 9885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d5789a8a6e7449d96a57c12166788bf [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:30.161500 9885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d5789a8a6e7449d96a57c12166788bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d5789a8a6e7449d96a57c12166788bf" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38977 } }
I20260928 08:57:30.161615 9885 leader_election.cc:302] T 00000000000000000000000000000000 P 8d5789a8a6e7449d96a57c12166788bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d5789a8a6e7449d96a57c12166788bf; no voters:
I20260928 08:57:30.161777 9885 leader_election.cc:288] T 00000000000000000000000000000000 P 8d5789a8a6e7449d96a57c12166788bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:57:30.161924 9889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d5789a8a6e7449d96a57c12166788bf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:57:30.162038 9885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d5789a8a6e7449d96a57c12166788bf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:57:30.162051 9889 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d5789a8a6e7449d96a57c12166788bf [term 1 LEADER]: Becoming Leader. State: Replica: 8d5789a8a6e7449d96a57c12166788bf, State: Running, Role: LEADER
I20260928 08:57:30.162137 9889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d5789a8a6e7449d96a57c12166788bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d5789a8a6e7449d96a57c12166788bf" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38977 } }
I20260928 08:57:30.162683 9890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d5789a8a6e7449d96a57c12166788bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d5789a8a6e7449d96a57c12166788bf. Latest consensus state: current_term: 1 leader_uuid: "8d5789a8a6e7449d96a57c12166788bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d5789a8a6e7449d96a57c12166788bf" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38977 } } }
I20260928 08:57:30.162771 9890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d5789a8a6e7449d96a57c12166788bf [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:30.162807 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 9817
I20260928 08:57:30.162911 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:57:30.163136 24223 external_mini_cluster.cc:1428] Setting key e5037e6a061930bef381a1b9aa7bd7aa
I20260928 08:57:30.163129 9888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d5789a8a6e7449d96a57c12166788bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d5789a8a6e7449d96a57c12166788bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d5789a8a6e7449d96a57c12166788bf" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38977 } } }
I20260928 08:57:30.163201 9888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d5789a8a6e7449d96a57c12166788bf [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:30.163784 9897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:57:30.164350 9897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:57:30.165953 9897 catalog_manager.cc:1347] Generated new cluster ID: 2a59fbb8a1c24727a8878d9c6ea966ff
I20260928 08:57:30.166002 9897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 08:57:30 dist-test-slave-pbjp krb5kdc[9804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:57:30.174299 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:30.163613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47414 (local address 127.23.167.254:38977)
0928 08:57:30.163770 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:30.163780 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:30.163820 (+ 40us) server_negotiation.cc:407] Connection header received
0928 08:57:30.163947 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:30.163956 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:30.164057 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:30.164211 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:30.164868 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.166140 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.167201 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.167820 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.167921 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:30.170668 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:30.170673 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:30.170673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:30.170688 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:30.172662 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:30.173039 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:30.173042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:30.173043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:30.173197 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:30.173372 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:30.173373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:30.173373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:30.173703 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:30.173853 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:30.173922 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 08:57:30.174071 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":55,"threads_started":1}
I20260928 08:57:30.175297 9897 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:57:30.175976 9897 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:57:30.184860 9897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d5789a8a6e7449d96a57c12166788bf: Generated new TSK 0
I20260928 08:57:30.185055 9897 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.167.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
I20260928 08:57:30.211612 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:0
--local_ip_for_outbound_sockets=127.23.167.193
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:38977
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:38329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:30.291929 9911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:30.292133 9911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:30.292155 9911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:30.293298 9911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:30.293344 9911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:30.293375 9911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:57:30.294786 9911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:38329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:38977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:30.295006 9911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:30.295218 9911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:30.300032 9916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:30.300125 9917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:30.300241 9911 server_base.cc:1034] running on GCE node
W20260928 08:57:30.300055 9919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:30.300506 9911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:30.300729 9911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:30.301903 9911 hybrid_clock.cc:648] HybridClock initialized: now 1790585850301884 us; error 27 us; skew 500 ppm
Sep 28 08:57:30 dist-test-slave-pbjp krb5kdc[9804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585850, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:30.304374 9911 init.cc:377] Logged in from keytab as kudu/127.23.167.193@KRBTEST.COM (short username kudu)
I20260928 08:57:30.304821 9911 webserver.cc:460] Webserver started at http://127.23.167.193:42357/ using document root <none> and password file <none>
I20260928 08:57:30.305033 9911 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:30.305126 9911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:30.305254 9911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:30.306335 9911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-0/data/instance:
uuid: "42f22908fe5848609ef6f7d950f2385e"
format_stamp: "Formatted at 2026-09-28 08:57:30 on dist-test-slave-pbjp"
server_key: "0e27bb2b7d8c8fd0c51e0056c8042991"
server_key_iv: "91d93e3e35dd7735eb16038d66797050"
server_key_version: "encryptionkey@0"
I20260928 08:57:30.306663 9911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance:
uuid: "42f22908fe5848609ef6f7d950f2385e"
format_stamp: "Formatted at 2026-09-28 08:57:30 on dist-test-slave-pbjp"
server_key: "0e27bb2b7d8c8fd0c51e0056c8042991"
server_key_iv: "91d93e3e35dd7735eb16038d66797050"
server_key_version: "encryptionkey@0"
I20260928 08:57:30.307901 9911 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 08:57:30.308578 9926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:30.308728 9911 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:57:30.308809 9911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "42f22908fe5848609ef6f7d950f2385e"
format_stamp: "Formatted at 2026-09-28 08:57:30 on dist-test-slave-pbjp"
server_key: "0e27bb2b7d8c8fd0c51e0056c8042991"
server_key_iv: "91d93e3e35dd7735eb16038d66797050"
server_key_version: "encryptionkey@0"
I20260928 08:57:30.308871 9911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:30.319851 9911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:30.321659 9911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:30.321798 9911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:30.322028 9911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:57:30.322352 9911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:57:30.322386 9911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:30.322417 9911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:57:30.322440 9911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:30.328297 9911 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:35915
I20260928 08:57:30.328339 10039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:35915 every 8 connection(s)
I20260928 08:57:30.328670 9911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
Sep 28 08:57:30 dist-test-slave-pbjp krb5kdc[9804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585850, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:57:30.336540 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 9911
I20260928 08:57:30.336664 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance
I20260928 08:57:30.336892 24223 external_mini_cluster.cc:1428] Setting key 240d910157a6a5faef342a7ce22e03bb
I20260928 08:57:30.337859 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:30.329331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:46081 (local address 127.23.167.254:38977)
0928 08:57:30.329382 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:30.329384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:30.329594 (+ 210us) server_negotiation.cc:407] Connection header received
0928 08:57:30.330098 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:30.330099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:30.330111 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:30.330149 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:30.331089 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.331606 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.332666 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.332864 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.332912 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:30.335069 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:30.335070 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:30.335071 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:30.335078 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:30.336322 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:30.336803 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:30.336804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:30.336805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:30.336861 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:30.337150 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:30.337151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:30.337152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:30.337280 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:30.337340 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:30.337640 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 08:57:30.337743 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 08:57:30.337977 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:30.329314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38977 (local address 127.23.167.193:46081)
0928 08:57:30.329471 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:30.329515 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:30.329998 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:30.330212 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:30.330215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:30.330799 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:57:30.331014 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:30.331020 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.331699 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.331700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:30.332593 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:30.332597 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.332936 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.332937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:30.333042 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:30.333568 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:57:30.333577 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:57:30.334964 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:57:30.336407 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:30.336416 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:30.336419 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:30.336731 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:30.336924 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:30.336925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:30.336925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:30.337093 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:30.337343 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:57:30.337351 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:57:30.337506 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 08:57:30.337605 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 08:57:30.337779 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":63,"threads_started":1}
I20260928 08:57:30.338660 10040 heartbeater.cc:347] Connected to a master server at 127.23.167.254:38977
I20260928 08:57:30.338760 10040 heartbeater.cc:464] Registering TS with master...
I20260928 08:57:30.338943 10040 heartbeater.cc:511] Master 127.23.167.254:38977 requested a full tablet report, sending...
I20260928 08:57:30.339488 9849 ts_manager.cc:194] Registered new tserver with Master: 42f22908fe5848609ef6f7d950f2385e (127.23.167.193:35915)
I20260928 08:57:30.340135 9849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.193@KRBTEST.COM'} at 127.23.167.193:46081
WARNING: no policy specified for kudu/127.23.167.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
I20260928 08:57:30.360059 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:0
--local_ip_for_outbound_sockets=127.23.167.194
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:38977
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:38329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:30.440420 10045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:30.440644 10045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:30.440670 10045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:30.441941 10045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:30.441994 10045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:30.442023 10045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:57:30.443538 10045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:38329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:38977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:30.443776 10045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:30.444007 10045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:30.448935 10051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:30.449028 10045 server_base.cc:1034] running on GCE node
W20260928 08:57:30.448995 10053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:30.448984 10050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:30.449347 10045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:30.449573 10045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:30.450709 10045 hybrid_clock.cc:648] HybridClock initialized: now 1790585850450699 us; error 30 us; skew 500 ppm
Sep 28 08:57:30 dist-test-slave-pbjp krb5kdc[9804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585850, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:30.453140 10045 init.cc:377] Logged in from keytab as kudu/127.23.167.194@KRBTEST.COM (short username kudu)
I20260928 08:57:30.453522 10045 webserver.cc:460] Webserver started at http://127.23.167.194:43145/ using document root <none> and password file <none>
I20260928 08:57:30.453696 10045 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:30.453735 10045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:30.453825 10045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:30.454804 10045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-1/data/instance:
uuid: "a2bfb5ae7e4a45508c0c0fc7e49886ef"
format_stamp: "Formatted at 2026-09-28 08:57:30 on dist-test-slave-pbjp"
server_key: "1f424492da1fdf7d7a7074e45a0fc219"
server_key_iv: "e8dacd7ae1c29ab628257a2caf219500"
server_key_version: "encryptionkey@0"
I20260928 08:57:30.455111 10045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance:
uuid: "a2bfb5ae7e4a45508c0c0fc7e49886ef"
format_stamp: "Formatted at 2026-09-28 08:57:30 on dist-test-slave-pbjp"
server_key: "1f424492da1fdf7d7a7074e45a0fc219"
server_key_iv: "e8dacd7ae1c29ab628257a2caf219500"
server_key_version: "encryptionkey@0"
I20260928 08:57:30.456238 10045 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:57:30.456861 10060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:30.457019 10045 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:30.457118 10045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "a2bfb5ae7e4a45508c0c0fc7e49886ef"
format_stamp: "Formatted at 2026-09-28 08:57:30 on dist-test-slave-pbjp"
server_key: "1f424492da1fdf7d7a7074e45a0fc219"
server_key_iv: "e8dacd7ae1c29ab628257a2caf219500"
server_key_version: "encryptionkey@0"
I20260928 08:57:30.457178 10045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:30.470961 10045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:30.473055 10045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:30.473228 10045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:30.473449 10045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:57:30.473739 10045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:57:30.473770 10045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:30.473802 10045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:57:30.473825 10045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:30.479270 10045 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:36705
I20260928 08:57:30.479365 10173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:36705 every 8 connection(s)
I20260928 08:57:30.479627 10045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
I20260928 08:57:30.484870 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 10045
I20260928 08:57:30.484977 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance
I20260928 08:57:30.485177 24223 external_mini_cluster.cc:1428] Setting key 35686eb8f035f557505a5ece7025e833
Sep 28 08:57:30 dist-test-slave-pbjp krb5kdc[9804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585850, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:57:30.488248 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:30.480297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:51095 (local address 127.23.167.254:38977)
0928 08:57:30.480349 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:30.480352 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:30.480531 (+ 179us) server_negotiation.cc:407] Connection header received
0928 08:57:30.481016 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:30.481017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:30.481027 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:30.481121 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:30.482006 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.482499 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.483450 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.483632 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.483675 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:30.485673 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:30.485675 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:30.485675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:30.485682 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:30.487015 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:30.487363 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:30.487364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:30.487365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:30.487431 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:30.487640 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:30.487641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:30.487641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:30.487780 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:30.487842 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:30.488118 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 08:57:30.488185 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 08:57:30.488456 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:30.480272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38977 (local address 127.23.167.194:51095)
0928 08:57:30.480414 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:30.480449 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:30.480925 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:30.481179 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:30.481182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:30.481726 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:57:30.481931 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:30.481936 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.482578 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.482578 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:30.483386 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:30.483389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.483690 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.483691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:30.483791 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:30.484226 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:57:30.484233 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:57:30.485594 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:57:30.487089 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:30.487095 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:30.487097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:30.487302 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:30.487484 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:30.487484 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:30.487484 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:30.487588 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:30.487839 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:57:30.487844 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:57:30.487991 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 08:57:30.488096 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 08:57:30.488247 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
I20260928 08:57:30.488899 10174 heartbeater.cc:347] Connected to a master server at 127.23.167.254:38977
I20260928 08:57:30.489002 10174 heartbeater.cc:464] Registering TS with master...
I20260928 08:57:30.489207 10174 heartbeater.cc:511] Master 127.23.167.254:38977 requested a full tablet report, sending...
I20260928 08:57:30.489578 9849 ts_manager.cc:194] Registered new tserver with Master: a2bfb5ae7e4a45508c0c0fc7e49886ef (127.23.167.194:36705)
I20260928 08:57:30.490058 9849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.194@KRBTEST.COM'} at 127.23.167.194:51095
WARNING: no policy specified for kudu/127.23.167.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
I20260928 08:57:30.509321 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:0
--local_ip_for_outbound_sockets=127.23.167.195
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:38977
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:38329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:30.592247 10179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:30.592437 10179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:30.592468 10179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:30.593658 10179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:30.593698 10179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:30.593731 10179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:57:30.595144 10179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:38329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:38977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:30.595359 10179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:30.595554 10179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:30.601225 10184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:30.601234 10187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:30.601234 10185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:30.601622 10179 server_base.cc:1034] running on GCE node
I20260928 08:57:30.601811 10179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:30.602031 10179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:30.603179 10179 hybrid_clock.cc:648] HybridClock initialized: now 1790585850603167 us; error 25 us; skew 500 ppm
Sep 28 08:57:30 dist-test-slave-pbjp krb5kdc[9804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585850, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:30.605633 10179 init.cc:377] Logged in from keytab as kudu/127.23.167.195@KRBTEST.COM (short username kudu)
I20260928 08:57:30.606032 10179 webserver.cc:460] Webserver started at http://127.23.167.195:43131/ using document root <none> and password file <none>
I20260928 08:57:30.606251 10179 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:30.606299 10179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:30.606423 10179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:30.607455 10179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-2/data/instance:
uuid: "3b027ae536314184a2e76bfc634d1c93"
format_stamp: "Formatted at 2026-09-28 08:57:30 on dist-test-slave-pbjp"
server_key: "0aa6a33c1e769de214041116adab58d9"
server_key_iv: "086e12b8c73b25a15e157d9f1940f37f"
server_key_version: "encryptionkey@0"
I20260928 08:57:30.607790 10179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance:
uuid: "3b027ae536314184a2e76bfc634d1c93"
format_stamp: "Formatted at 2026-09-28 08:57:30 on dist-test-slave-pbjp"
server_key: "0aa6a33c1e769de214041116adab58d9"
server_key_iv: "086e12b8c73b25a15e157d9f1940f37f"
server_key_version: "encryptionkey@0"
I20260928 08:57:30.609129 10179 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 08:57:30.609910 10194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:30.610076 10179 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:57:30.610152 10179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "3b027ae536314184a2e76bfc634d1c93"
format_stamp: "Formatted at 2026-09-28 08:57:30 on dist-test-slave-pbjp"
server_key: "0aa6a33c1e769de214041116adab58d9"
server_key_iv: "086e12b8c73b25a15e157d9f1940f37f"
server_key_version: "encryptionkey@0"
I20260928 08:57:30.610221 10179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:30.619987 10179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:30.622008 10179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:30.622145 10179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:30.622365 10179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:57:30.622658 10179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:57:30.622691 10179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:30.622720 10179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:57:30.622743 10179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:30.628259 10179 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:41499
I20260928 08:57:30.628324 10307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:41499 every 8 connection(s)
I20260928 08:57:30.628609 10179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
I20260928 08:57:30.634207 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 10179
I20260928 08:57:30.634300 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance
I20260928 08:57:30.634459 24223 external_mini_cluster.cc:1428] Setting key 208c8916345cb7c83e2e3b3c878172f3
Sep 28 08:57:30 dist-test-slave-pbjp krb5kdc[9804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585850, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:57:30.637548 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:30.629328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:34969 (local address 127.23.167.254:38977)
0928 08:57:30.629379 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:30.629381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:30.629599 (+ 218us) server_negotiation.cc:407] Connection header received
0928 08:57:30.630077 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:30.630078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:30.630088 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:30.630125 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:30.631062 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.631553 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.632580 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.632757 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.632802 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:30.635154 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:30.635155 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:30.635156 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:30.635163 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:30.636377 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:30.636733 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:30.636734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:30.636734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:30.636777 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:30.636962 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:30.636962 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:30.636962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:30.637093 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:30.637150 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:30.637429 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 08:57:30.637484 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 08:57:30.637758 10310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:30.629328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38977 (local address 127.23.167.195:34969)
0928 08:57:30.629483 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:30.629521 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:30.629989 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:30.630182 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:30.630185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:30.630748 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:57:30.630976 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:30.630982 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.631656 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.631657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:30.632508 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:30.632511 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.632817 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.632818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:30.632935 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:30.633488 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:57:30.633496 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:57:30.635055 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:57:30.636450 (+ 1395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:30.636456 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:30.636458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:30.636671 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:30.636822 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:30.636822 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:30.636823 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:30.636919 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:30.637149 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:57:30.637156 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:57:30.637286 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 08:57:30.637411 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 08:57:30.637535 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260928 08:57:30.638177 10308 heartbeater.cc:347] Connected to a master server at 127.23.167.254:38977
I20260928 08:57:30.638257 10308 heartbeater.cc:464] Registering TS with master...
I20260928 08:57:30.638411 10308 heartbeater.cc:511] Master 127.23.167.254:38977 requested a full tablet report, sending...
I20260928 08:57:30.638729 9849 ts_manager.cc:194] Registered new tserver with Master: 3b027ae536314184a2e76bfc634d1c93 (127.23.167.195:41499)
I20260928 08:57:30.639056 9849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.195@KRBTEST.COM'} at 127.23.167.195:34969
I20260928 08:57:30.646162 24223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:57:30.653612 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:30.647502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47418 (local address 127.23.167.254:38977)
0928 08:57:30.647568 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:30.647570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:30.647650 (+ 80us) server_negotiation.cc:407] Connection header received
0928 08:57:30.647721 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:30.647722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:30.647732 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:30.647773 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:30.648309 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.648802 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.649535 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.649739 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.649794 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:30.651009 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:30.651011 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:30.651011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:30.651019 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:30.652315 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:30.652778 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:30.652779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:30.652779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:30.652835 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:30.653042 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:30.653043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:30.653043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:30.653215 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:30.653311 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:30.653455 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 08:57:30.653517 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 08:57:30.655256 9849 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:57:30.656888 9849 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:57:30.663487 10318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:30.659034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51514 (local address 127.23.167.193:35915)
0928 08:57:30.659419 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:30.659423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:30.659432 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:57:30.659472 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:30.659474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:30.659487 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:30.659567 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:57:30.659879 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.660740 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.662207 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.663050 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.663140 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:30.663208 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:30.663332 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 08:57:30.663378 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":97,"threads_started":1}
I20260928 08:57:30.663527 10317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:30.659138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:36705 (local address 127.0.0.1:60206)
0928 08:57:30.659455 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:30.659464 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:30.659512 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:30.659784 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:30.659784 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:30.659786 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:57:30.659812 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:30.659813 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.661181 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.661183 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:30.662251 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:30.662253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.663287 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.663288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:30.663352 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:30.663357 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 08:57:30.663403 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 08:57:30.663462 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":38,"threads_started":1}
I20260928 08:57:30.663821 10321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:30.659216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60206 (local address 127.23.167.194:36705)
0928 08:57:30.659384 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:30.659389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:30.659523 (+ 134us) server_negotiation.cc:407] Connection header received
0928 08:57:30.659566 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:30.659568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:30.659583 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:30.659655 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:57:30.659891 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.661083 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.662327 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.663147 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.663207 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:30.663253 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:30.663723 (+ 470us) server_negotiation.cc:299] Negotiation successful
0928 08:57:30.663760 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":66,"threads_started":1}
I20260928 08:57:30.663846 10320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:30.658921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:35915 (local address 127.0.0.1:51514)
0928 08:57:30.659241 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:30.659283 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:30.659364 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:30.659686 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:30.659689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:30.659692 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:57:30.659798 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:30.659800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.660816 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.660818 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:30.662119 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:30.662123 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.663147 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.663148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:30.663224 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:30.663240 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 08:57:30.663309 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 08:57:30.663363 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":23,"thread_start_us":65,"threads_started":1}
I20260928 08:57:30.664589 10316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:30.658905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:41499 (local address 127.0.0.1:56222)
0928 08:57:30.659240 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:30.659283 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:30.659364 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:30.659737 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:30.659738 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:30.659740 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:57:30.659810 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:30.659812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.662321 (+ 2509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.662321 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:30.663114 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:30.663117 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.664372 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.664373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:30.664433 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:30.664436 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:57:30.664481 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 08:57:30.664523 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":2,"thread_start_us":65,"threads_started":1}
I20260928 08:57:30.664595 9974 tablet_service.cc:1467] Processing CreateTablet for tablet 2283a33017b54613b0bc0e8bf1053cff (DEFAULT_TABLE table=test-table [id=5ec851f52e624a6097b6b50854879ac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:57:30.664594 10108 tablet_service.cc:1467] Processing CreateTablet for tablet 2283a33017b54613b0bc0e8bf1053cff (DEFAULT_TABLE table=test-table [id=5ec851f52e624a6097b6b50854879ac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:57:30.664953 10319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:30.659024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56222 (local address 127.23.167.195:41499)
0928 08:57:30.659518 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:30.659522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:30.659532 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:57:30.659577 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:30.659580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:30.659594 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:30.659660 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:57:30.659887 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.661083 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.663203 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.664225 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.664286 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:30.664337 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:30.664834 (+ 497us) server_negotiation.cc:299] Negotiation successful
0928 08:57:30.664877 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":109,"threads_started":1}
I20260928 08:57:30.664973 10108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2283a33017b54613b0bc0e8bf1053cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:30.664983 9974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2283a33017b54613b0bc0e8bf1053cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:30.665691 10242 tablet_service.cc:1467] Processing CreateTablet for tablet 2283a33017b54613b0bc0e8bf1053cff (DEFAULT_TABLE table=test-table [id=5ec851f52e624a6097b6b50854879ac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:57:30.666000 10242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2283a33017b54613b0bc0e8bf1053cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:30.667714 10322 tablet_bootstrap.cc:492] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef: Bootstrap starting.
I20260928 08:57:30.668170 10323 tablet_bootstrap.cc:492] T 2283a33017b54613b0bc0e8bf1053cff P 42f22908fe5848609ef6f7d950f2385e: Bootstrap starting.
I20260928 08:57:30.668412 10322 tablet_bootstrap.cc:654] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:30.668943 10324 tablet_bootstrap.cc:492] T 2283a33017b54613b0bc0e8bf1053cff P 3b027ae536314184a2e76bfc634d1c93: Bootstrap starting.
I20260928 08:57:30.669090 10323 tablet_bootstrap.cc:654] T 2283a33017b54613b0bc0e8bf1053cff P 42f22908fe5848609ef6f7d950f2385e: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:30.669142 10322 log.cc:824] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:30.669399 10323 log.cc:824] T 2283a33017b54613b0bc0e8bf1053cff P 42f22908fe5848609ef6f7d950f2385e: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:30.669595 10324 tablet_bootstrap.cc:654] T 2283a33017b54613b0bc0e8bf1053cff P 3b027ae536314184a2e76bfc634d1c93: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:30.669932 10322 tablet_bootstrap.cc:492] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef: No bootstrap required, opened a new log
I20260928 08:57:30.669976 10322 ts_tablet_manager.cc:1397] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 08:57:30.670068 10324 log.cc:824] T 2283a33017b54613b0bc0e8bf1053cff P 3b027ae536314184a2e76bfc634d1c93: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:30.670138 10323 tablet_bootstrap.cc:492] T 2283a33017b54613b0bc0e8bf1053cff P 42f22908fe5848609ef6f7d950f2385e: No bootstrap required, opened a new log
I20260928 08:57:30.670188 10323 ts_tablet_manager.cc:1397] T 2283a33017b54613b0bc0e8bf1053cff P 42f22908fe5848609ef6f7d950f2385e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:57:30.670774 10324 tablet_bootstrap.cc:492] T 2283a33017b54613b0bc0e8bf1053cff P 3b027ae536314184a2e76bfc634d1c93: No bootstrap required, opened a new log
I20260928 08:57:30.670827 10324 ts_tablet_manager.cc:1397] T 2283a33017b54613b0bc0e8bf1053cff P 3b027ae536314184a2e76bfc634d1c93: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:57:30.671511 10322 raft_consensus.cc:348] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b027ae536314184a2e76bfc634d1c93" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 41499 } } peers { permanent_uuid: "42f22908fe5848609ef6f7d950f2385e" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35915 } } peers { permanent_uuid: "a2bfb5ae7e4a45508c0c0fc7e49886ef" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36705 } }
I20260928 08:57:30.671643 10322 raft_consensus.cc:374] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:30.671662 10322 raft_consensus.cc:729] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2bfb5ae7e4a45508c0c0fc7e49886ef, State: Initialized, Role: FOLLOWER
I20260928 08:57:30.671752 10322 consensus_queue.cc:260] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b027ae536314184a2e76bfc634d1c93" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 41499 } } peers { permanent_uuid: "42f22908fe5848609ef6f7d950f2385e" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35915 } } peers { permanent_uuid: "a2bfb5ae7e4a45508c0c0fc7e49886ef" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36705 } }
I20260928 08:57:30.672001 10322 ts_tablet_manager.cc:1428] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 08:57:30.672026 10174 heartbeater.cc:503] Master 127.23.167.254:38977 was elected leader, sending a full tablet report...
I20260928 08:57:30.671954 10323 raft_consensus.cc:348] T 2283a33017b54613b0bc0e8bf1053cff P 42f22908fe5848609ef6f7d950f2385e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b027ae536314184a2e76bfc634d1c93" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 41499 } } peers { permanent_uuid: "42f22908fe5848609ef6f7d950f2385e" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35915 } } peers { permanent_uuid: "a2bfb5ae7e4a45508c0c0fc7e49886ef" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36705 } }
I20260928 08:57:30.672127 10323 raft_consensus.cc:374] T 2283a33017b54613b0bc0e8bf1053cff P 42f22908fe5848609ef6f7d950f2385e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:30.672168 10323 raft_consensus.cc:729] T 2283a33017b54613b0bc0e8bf1053cff P 42f22908fe5848609ef6f7d950f2385e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42f22908fe5848609ef6f7d950f2385e, State: Initialized, Role: FOLLOWER
I20260928 08:57:30.672272 10323 consensus_queue.cc:260] T 2283a33017b54613b0bc0e8bf1053cff P 42f22908fe5848609ef6f7d950f2385e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b027ae536314184a2e76bfc634d1c93" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 41499 } } peers { permanent_uuid: "42f22908fe5848609ef6f7d950f2385e" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35915 } } peers { permanent_uuid: "a2bfb5ae7e4a45508c0c0fc7e49886ef" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36705 } }
I20260928 08:57:30.672531 10323 ts_tablet_manager.cc:1428] T 2283a33017b54613b0bc0e8bf1053cff P 42f22908fe5848609ef6f7d950f2385e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 08:57:30.672571 10040 heartbeater.cc:503] Master 127.23.167.254:38977 was elected leader, sending a full tablet report...
I20260928 08:57:30.672683 10324 raft_consensus.cc:348] T 2283a33017b54613b0bc0e8bf1053cff P 3b027ae536314184a2e76bfc634d1c93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b027ae536314184a2e76bfc634d1c93" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 41499 } } peers { permanent_uuid: "42f22908fe5848609ef6f7d950f2385e" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35915 } } peers { permanent_uuid: "a2bfb5ae7e4a45508c0c0fc7e49886ef" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36705 } }
I20260928 08:57:30.673156 10324 raft_consensus.cc:374] T 2283a33017b54613b0bc0e8bf1053cff P 3b027ae536314184a2e76bfc634d1c93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:30.673203 10324 raft_consensus.cc:729] T 2283a33017b54613b0bc0e8bf1053cff P 3b027ae536314184a2e76bfc634d1c93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b027ae536314184a2e76bfc634d1c93, State: Initialized, Role: FOLLOWER
I20260928 08:57:30.673419 10324 consensus_queue.cc:260] T 2283a33017b54613b0bc0e8bf1053cff P 3b027ae536314184a2e76bfc634d1c93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b027ae536314184a2e76bfc634d1c93" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 41499 } } peers { permanent_uuid: "42f22908fe5848609ef6f7d950f2385e" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35915 } } peers { permanent_uuid: "a2bfb5ae7e4a45508c0c0fc7e49886ef" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36705 } }
I20260928 08:57:30.673686 10324 ts_tablet_manager.cc:1428] T 2283a33017b54613b0bc0e8bf1053cff P 3b027ae536314184a2e76bfc634d1c93: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260928 08:57:30.673710 10308 heartbeater.cc:503] Master 127.23.167.254:38977 was elected leader, sending a full tablet report...
W20260928 08:57:30.730207 10175 tablet.cc:2369] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:57:30.829365 10041 tablet.cc:2369] T 2283a33017b54613b0bc0e8bf1053cff P 42f22908fe5848609ef6f7d950f2385e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:57:30.879163 10309 tablet.cc:2369] T 2283a33017b54613b0bc0e8bf1053cff P 3b027ae536314184a2e76bfc634d1c93: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:57:30.889560 10328 raft_consensus.cc:482] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:57:30.889683 10328 raft_consensus.cc:504] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b027ae536314184a2e76bfc634d1c93" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 41499 } } peers { permanent_uuid: "42f22908fe5848609ef6f7d950f2385e" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35915 } } peers { permanent_uuid: "a2bfb5ae7e4a45508c0c0fc7e49886ef" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36705 } }
I20260928 08:57:30.890007 10328 leader_election.cc:288] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b027ae536314184a2e76bfc634d1c93 (127.23.167.195:41499), 42f22908fe5848609ef6f7d950f2385e (127.23.167.193:35915)
I20260928 08:57:30.893805 10318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:30.890265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:58283 (local address 127.23.167.193:35915)
0928 08:57:30.890305 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:30.890307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:30.890455 (+ 148us) server_negotiation.cc:407] Connection header received
0928 08:57:30.890487 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:30.890488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:30.890495 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:30.890534 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:57:30.890675 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.891408 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.892849 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.893432 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.893506 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:30.893515 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:30.893693 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 08:57:30.893730 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 08:57:30.893805 10319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:30.890252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:39157 (local address 127.23.167.195:41499)
0928 08:57:30.890300 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:30.890302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:30.890338 (+ 36us) server_negotiation.cc:407] Connection header received
0928 08:57:30.890365 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:30.890366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:30.890375 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:30.890420 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:57:30.890609 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.891365 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.892863 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.893446 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.893507 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:30.893515 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:30.893693 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 08:57:30.893730 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 08:57:30.893824 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:30.890186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:41499 (local address 127.23.167.194:39157)
0928 08:57:30.890243 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:30.890254 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:30.890294 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:30.890442 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:30.890443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:30.890444 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:57:30.890545 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:30.890547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.891451 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.891452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:30.892756 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:30.892760 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.893527 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.893528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:30.893628 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:30.893630 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 08:57:30.893683 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 08:57:30.893732 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":55}
I20260928 08:57:30.893831 10331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:30.890195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:35915 (local address 127.23.167.194:58283)
0928 08:57:30.890384 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:30.890391 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:30.890441 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:30.890553 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:30.890553 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:30.890555 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:57:30.890616 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:30.890617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.891474 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.891475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:30.892756 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:30.892760 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.893517 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.893518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:30.893622 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:30.893627 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 08:57:30.893683 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 08:57:30.893730 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":70,"thread_start_us":100,"threads_started":1}
I20260928 08:57:30.894176 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2283a33017b54613b0bc0e8bf1053cff" candidate_uuid: "a2bfb5ae7e4a45508c0c0fc7e49886ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b027ae536314184a2e76bfc634d1c93" is_pre_election: true
I20260928 08:57:30.894330 10262 raft_consensus.cc:2454] T 2283a33017b54613b0bc0e8bf1053cff P 3b027ae536314184a2e76bfc634d1c93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2bfb5ae7e4a45508c0c0fc7e49886ef in term 0.
I20260928 08:57:30.894419 9994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2283a33017b54613b0bc0e8bf1053cff" candidate_uuid: "a2bfb5ae7e4a45508c0c0fc7e49886ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42f22908fe5848609ef6f7d950f2385e" is_pre_election: true
I20260928 08:57:30.894560 9994 raft_consensus.cc:2454] T 2283a33017b54613b0bc0e8bf1053cff P 42f22908fe5848609ef6f7d950f2385e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2bfb5ae7e4a45508c0c0fc7e49886ef in term 0.
I20260928 08:57:30.894564 10061 leader_election.cc:302] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b027ae536314184a2e76bfc634d1c93, a2bfb5ae7e4a45508c0c0fc7e49886ef; no voters:
I20260928 08:57:30.894668 10328 raft_consensus.cc:2781] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:57:30.894724 10328 raft_consensus.cc:482] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:57:30.894752 10328 raft_consensus.cc:3037] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:30.895404 10328 raft_consensus.cc:504] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b027ae536314184a2e76bfc634d1c93" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 41499 } } peers { permanent_uuid: "42f22908fe5848609ef6f7d950f2385e" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35915 } } peers { permanent_uuid: "a2bfb5ae7e4a45508c0c0fc7e49886ef" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36705 } }
I20260928 08:57:30.895515 10328 leader_election.cc:288] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef [CANDIDATE]: Term 1 election: Requested vote from peers 3b027ae536314184a2e76bfc634d1c93 (127.23.167.195:41499), 42f22908fe5848609ef6f7d950f2385e (127.23.167.193:35915)
I20260928 08:57:30.895684 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2283a33017b54613b0bc0e8bf1053cff" candidate_uuid: "a2bfb5ae7e4a45508c0c0fc7e49886ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b027ae536314184a2e76bfc634d1c93"
I20260928 08:57:30.895725 9994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2283a33017b54613b0bc0e8bf1053cff" candidate_uuid: "a2bfb5ae7e4a45508c0c0fc7e49886ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42f22908fe5848609ef6f7d950f2385e"
I20260928 08:57:30.895780 9994 raft_consensus.cc:3037] T 2283a33017b54613b0bc0e8bf1053cff P 42f22908fe5848609ef6f7d950f2385e [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:30.895766 10262 raft_consensus.cc:3037] T 2283a33017b54613b0bc0e8bf1053cff P 3b027ae536314184a2e76bfc634d1c93 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:30.896413 9994 raft_consensus.cc:2454] T 2283a33017b54613b0bc0e8bf1053cff P 42f22908fe5848609ef6f7d950f2385e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2bfb5ae7e4a45508c0c0fc7e49886ef in term 1.
I20260928 08:57:30.896489 10262 raft_consensus.cc:2454] T 2283a33017b54613b0bc0e8bf1053cff P 3b027ae536314184a2e76bfc634d1c93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2bfb5ae7e4a45508c0c0fc7e49886ef in term 1.
I20260928 08:57:30.896597 10062 leader_election.cc:302] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42f22908fe5848609ef6f7d950f2385e, a2bfb5ae7e4a45508c0c0fc7e49886ef; no voters:
I20260928 08:57:30.896708 10328 raft_consensus.cc:2781] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:57:30.896870 10328 raft_consensus.cc:686] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef [term 1 LEADER]: Becoming Leader. State: Replica: a2bfb5ae7e4a45508c0c0fc7e49886ef, State: Running, Role: LEADER
I20260928 08:57:30.896991 10328 consensus_queue.cc:237] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b027ae536314184a2e76bfc634d1c93" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 41499 } } peers { permanent_uuid: "42f22908fe5848609ef6f7d950f2385e" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35915 } } peers { permanent_uuid: "a2bfb5ae7e4a45508c0c0fc7e49886ef" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36705 } }
I20260928 08:57:30.897812 9849 catalog_manager.cc:5495] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef reported cstate change: term changed from 0 to 1, leader changed from <none> to a2bfb5ae7e4a45508c0c0fc7e49886ef (127.23.167.194). New cstate: current_term: 1 leader_uuid: "a2bfb5ae7e4a45508c0c0fc7e49886ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b027ae536314184a2e76bfc634d1c93" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42f22908fe5848609ef6f7d950f2385e" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2bfb5ae7e4a45508c0c0fc7e49886ef" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 36705 } health_report { overall_health: HEALTHY } } }
I20260928 08:57:30.945493 10321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:30.943330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60208 (local address 127.23.167.194:36705)
0928 08:57:30.943387 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:30.943389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:30.943397 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:57:30.943433 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:30.943434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:30.943452 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:30.943520 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:57:30.943653 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.944138 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.944834 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:30.945015 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:30.945075 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:30.945205 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:57:30.945292 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:57:30.945372 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:30.945392 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 08:57:30.945402 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 08:57:30.947440 9994 raft_consensus.cc:1261] T 2283a33017b54613b0bc0e8bf1053cff P 42f22908fe5848609ef6f7d950f2385e [term 1 FOLLOWER]: Refusing update from remote peer a2bfb5ae7e4a45508c0c0fc7e49886ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:57:30.947454 10262 raft_consensus.cc:1261] T 2283a33017b54613b0bc0e8bf1053cff P 3b027ae536314184a2e76bfc634d1c93 [term 1 FOLLOWER]: Refusing update from remote peer a2bfb5ae7e4a45508c0c0fc7e49886ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:57:30.947825 10328 consensus_queue.cc:1035] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b027ae536314184a2e76bfc634d1c93" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:57:30.948016 10332 consensus_queue.cc:1035] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "42f22908fe5848609ef6f7d950f2385e" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 35915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:57:30.950544 10337 mvcc.cc:205] Tried to move back new op lower bound from 7334239645477896192 to 7334239645274685440. Current Snapshot: MvccSnapshot[applied={T|T < 7334239645477896192}]
I20260928 08:57:30.951359 10336 mvcc.cc:205] Tried to move back new op lower bound from 7334239645477896192 to 7334239645274685440. Current Snapshot: MvccSnapshot[applied={T|T < 7334239645477896192}]
I20260928 08:57:30.953102 9848 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:57:30.953212 9848 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:57:30.953982 9848 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d5789a8a6e7449d96a57c12166788bf: Sending DeleteTablet for 3 replicas of tablet 2283a33017b54613b0bc0e8bf1053cff
I20260928 08:57:30.954308 10242 tablet_service.cc:1514] Processing DeleteTablet for tablet 2283a33017b54613b0bc0e8bf1053cff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:57:30 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:56222
I20260928 08:57:30.954332 9974 tablet_service.cc:1514] Processing DeleteTablet for tablet 2283a33017b54613b0bc0e8bf1053cff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:57:30 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:51514
I20260928 08:57:30.954591 10345 tablet_replica.cc:330] T 2283a33017b54613b0bc0e8bf1053cff P 42f22908fe5848609ef6f7d950f2385e: stopping tablet replica
I20260928 08:57:30.954676 10108 tablet_service.cc:1514] Processing DeleteTablet for tablet 2283a33017b54613b0bc0e8bf1053cff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:57:30 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:60206
I20260928 08:57:30.954746 10345 raft_consensus.cc:2229] T 2283a33017b54613b0bc0e8bf1053cff P 42f22908fe5848609ef6f7d950f2385e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:57:30.954864 10345 raft_consensus.cc:2258] T 2283a33017b54613b0bc0e8bf1053cff P 42f22908fe5848609ef6f7d950f2385e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:57:30.954957 10344 tablet_replica.cc:330] T 2283a33017b54613b0bc0e8bf1053cff P 3b027ae536314184a2e76bfc634d1c93: stopping tablet replica
I20260928 08:57:30.955040 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 9911
I20260928 08:57:30.955559 10346 tablet_replica.cc:330] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef: stopping tablet replica
I20260928 08:57:30.955641 10346 raft_consensus.cc:2229] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:57:30.955721 10346 raft_consensus.cc:2258] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:57:30.955847 10344 raft_consensus.cc:2229] T 2283a33017b54613b0bc0e8bf1053cff P 3b027ae536314184a2e76bfc634d1c93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:57:30.955953 10344 raft_consensus.cc:2258] T 2283a33017b54613b0bc0e8bf1053cff P 3b027ae536314184a2e76bfc634d1c93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:57:30.956394 10344 ts_tablet_manager.cc:1905] T 2283a33017b54613b0bc0e8bf1053cff P 3b027ae536314184a2e76bfc634d1c93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:57:30.957096 10346 ts_tablet_manager.cc:1905] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:57:30.958330 10344 ts_tablet_manager.cc:1918] T 2283a33017b54613b0bc0e8bf1053cff P 3b027ae536314184a2e76bfc634d1c93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:57:30.958428 10344 log.cc:1196] T 2283a33017b54613b0bc0e8bf1053cff P 3b027ae536314184a2e76bfc634d1c93: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-2/wal/wals/2283a33017b54613b0bc0e8bf1053cff
I20260928 08:57:30.958702 10346 ts_tablet_manager.cc:1918] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:57:30.958781 10344 ts_tablet_manager.cc:1939] T 2283a33017b54613b0bc0e8bf1053cff P 3b027ae536314184a2e76bfc634d1c93: Deleting consensus metadata
I20260928 08:57:30.959052 10346 log.cc:1196] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585589999956-24223-0/minicluster-data/ts-1/wal/wals/2283a33017b54613b0bc0e8bf1053cff
I20260928 08:57:30.959213 9833 catalog_manager.cc:4841] TS 3b027ae536314184a2e76bfc634d1c93 (127.23.167.195:41499): tablet 2283a33017b54613b0bc0e8bf1053cff (table test-table [id=5ec851f52e624a6097b6b50854879ac5]) successfully deleted
I20260928 08:57:30.959372 10346 ts_tablet_manager.cc:1939] T 2283a33017b54613b0bc0e8bf1053cff P a2bfb5ae7e4a45508c0c0fc7e49886ef: Deleting consensus metadata
I20260928 08:57:30.959761 9834 catalog_manager.cc:4841] TS a2bfb5ae7e4a45508c0c0fc7e49886ef (127.23.167.194:36705): tablet 2283a33017b54613b0bc0e8bf1053cff (table test-table [id=5ec851f52e624a6097b6b50854879ac5]) successfully deleted
W20260928 08:57:30.963033 9834 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.167.193:35915 (error 108)
I20260928 08:57:30.963243 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 10045
I20260928 08:57:30.964684 10320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:30.964541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:35915 (local address 127.0.0.1:51518)
0928 08:57:30.964607 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:30.964650 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.193:35915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 08:57:30.964792 9834 catalog_manager.cc:4584] TS 42f22908fe5848609ef6f7d950f2385e (127.23.167.193:35915): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2283a33017b54613b0bc0e8bf1053cff: Network error: Client connection negotiation failed: client connection to 127.23.167.193:35915: connect: Connection refused (error 111)
I20260928 08:57:30.967433 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 10179
I20260928 08:57:30.971575 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 9817
2026-09-28T08:57:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2012 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T08:57:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:57:30Z Disabled control of system clock
I20260928 08:57:30.996338 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:46193
--webserver_interface=127.23.167.254
--webserver_port=0
--builtin_ntp_servers=127.23.167.212:45157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:46193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 08:57:31.080618 10351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:31.080912 10351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:31.080945 10351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:31.082535 10351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:57:31.082641 10351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:31.082674 10351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:31.082695 10351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:57:31.082710 10351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:57:31.084647 10351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:45157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:46193
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:46193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:31.084937 10351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:31.085318 10351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:31.090428 10357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:31.090452 10359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:31.090581 10356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:31.091579 10351 server_base.cc:1034] running on GCE node
I20260928 08:57:31.091812 10351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:31.092077 10351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:31.093218 10351 hybrid_clock.cc:648] HybridClock initialized: now 1790585851093200 us; error 35 us; skew 500 ppm
I20260928 08:57:31.094688 10351 webserver.cc:460] Webserver started at http://127.23.167.254:38383/ using document root <none> and password file <none>
I20260928 08:57:31.094986 10351 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:31.095036 10351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:31.095166 10351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:31.096274 10351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "545dc129a06c43449a55fa4234ee710f"
format_stamp: "Formatted at 2026-09-28 08:57:31 on dist-test-slave-pbjp"
server_key: "250ac86620b7850e9ca3c0c21a3608df"
server_key_iv: "be8f11e9a9fbb2077d283903af36ce3c"
server_key_version: "encryptionkey@0"
I20260928 08:57:31.096621 10351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "545dc129a06c43449a55fa4234ee710f"
format_stamp: "Formatted at 2026-09-28 08:57:31 on dist-test-slave-pbjp"
server_key: "250ac86620b7850e9ca3c0c21a3608df"
server_key_iv: "be8f11e9a9fbb2077d283903af36ce3c"
server_key_version: "encryptionkey@0"
I20260928 08:57:31.098268 10351 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 08:57:31.099133 10365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:31.099294 10351 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:57:31.099390 10351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "545dc129a06c43449a55fa4234ee710f"
format_stamp: "Formatted at 2026-09-28 08:57:31 on dist-test-slave-pbjp"
server_key: "250ac86620b7850e9ca3c0c21a3608df"
server_key_iv: "be8f11e9a9fbb2077d283903af36ce3c"
server_key_version: "encryptionkey@0"
I20260928 08:57:31.099512 10351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:31.121186 10351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:31.121562 10351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:31.121701 10351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:31.125442 10351 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:46193
I20260928 08:57:31.125484 10417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:46193 every 8 connection(s)
I20260928 08:57:31.125813 10351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:57:31.126782 10418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:31.128999 10418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 545dc129a06c43449a55fa4234ee710f: Bootstrap starting.
I20260928 08:57:31.129578 10418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 545dc129a06c43449a55fa4234ee710f: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:31.129834 10418 log.cc:824] T 00000000000000000000000000000000 P 545dc129a06c43449a55fa4234ee710f: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:31.130227 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 10351
I20260928 08:57:31.130326 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:57:31.130545 10418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 545dc129a06c43449a55fa4234ee710f: No bootstrap required, opened a new log
I20260928 08:57:31.130571 24223 external_mini_cluster.cc:1428] Setting key 0f20e24c0a9daf24b689eae8301c22f5
I20260928 08:57:31.133015 10418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 545dc129a06c43449a55fa4234ee710f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545dc129a06c43449a55fa4234ee710f" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46193 } }
I20260928 08:57:31.133211 10418 raft_consensus.cc:374] T 00000000000000000000000000000000 P 545dc129a06c43449a55fa4234ee710f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:31.133257 10418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 545dc129a06c43449a55fa4234ee710f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 545dc129a06c43449a55fa4234ee710f, State: Initialized, Role: FOLLOWER
I20260928 08:57:31.133383 10418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 545dc129a06c43449a55fa4234ee710f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545dc129a06c43449a55fa4234ee710f" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46193 } }
I20260928 08:57:31.133457 10418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 545dc129a06c43449a55fa4234ee710f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:57:31.133493 10418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 545dc129a06c43449a55fa4234ee710f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:57:31.133538 10418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 545dc129a06c43449a55fa4234ee710f [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:31.134184 10418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 545dc129a06c43449a55fa4234ee710f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545dc129a06c43449a55fa4234ee710f" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46193 } }
I20260928 08:57:31.134274 10418 leader_election.cc:302] T 00000000000000000000000000000000 P 545dc129a06c43449a55fa4234ee710f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 545dc129a06c43449a55fa4234ee710f; no voters:
I20260928 08:57:31.134416 10418 leader_election.cc:288] T 00000000000000000000000000000000 P 545dc129a06c43449a55fa4234ee710f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:57:31.134443 10423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 545dc129a06c43449a55fa4234ee710f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:57:31.134563 10423 raft_consensus.cc:686] T 00000000000000000000000000000000 P 545dc129a06c43449a55fa4234ee710f [term 1 LEADER]: Becoming Leader. State: Replica: 545dc129a06c43449a55fa4234ee710f, State: Running, Role: LEADER
I20260928 08:57:31.134672 10418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 545dc129a06c43449a55fa4234ee710f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:57:31.134696 10423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 545dc129a06c43449a55fa4234ee710f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545dc129a06c43449a55fa4234ee710f" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46193 } }
I20260928 08:57:31.135032 10423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 545dc129a06c43449a55fa4234ee710f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 545dc129a06c43449a55fa4234ee710f. Latest consensus state: current_term: 1 leader_uuid: "545dc129a06c43449a55fa4234ee710f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545dc129a06c43449a55fa4234ee710f" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46193 } } }
I20260928 08:57:31.135048 10425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 545dc129a06c43449a55fa4234ee710f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "545dc129a06c43449a55fa4234ee710f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545dc129a06c43449a55fa4234ee710f" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46193 } } }
I20260928 08:57:31.135205 10425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 545dc129a06c43449a55fa4234ee710f [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:31.135190 10423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 545dc129a06c43449a55fa4234ee710f [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:31.135499 10432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:57:31.136251 10432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:57:31.137266 10422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:31.131060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59268 (local address 127.23.167.254:46193)
0928 08:57:31.131240 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:31.131248 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:31.131272 (+ 24us) server_negotiation.cc:407] Connection header received
0928 08:57:31.131755 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:31.131761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:31.131819 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:31.131940 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:31.132135 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.133007 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.134049 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.134559 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.135025 (+ 466us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:31.136288 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:31.136293 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:31.136293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:31.136487 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:31.136521 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:31.136532 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:31.136588 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:31.136932 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 08:57:31.137009 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":59,"threads_started":1}
I20260928 08:57:31.137884 10432 catalog_manager.cc:1347] Generated new cluster ID: 859ac817bb6041caa4256f3252e3b8de
I20260928 08:57:31.137934 10432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:57:31.154474 10432 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:57:31.155042 10432 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:57:31.158212 10432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 545dc129a06c43449a55fa4234ee710f: Generated new TSK 0
I20260928 08:57:31.158384 10432 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:57:31.162935 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:0
--local_ip_for_outbound_sockets=127.23.167.193
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:46193
--builtin_ntp_servers=127.23.167.212:45157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 08:57:31.247650 10442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:31.247862 10442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:31.247884 10442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:31.249125 10442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:31.249168 10442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:31.249193 10442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:57:31.250561 10442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:45157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:46193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:31.250787 10442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:31.251087 10442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 08:57:31.257009 10442 server_base.cc:1034] running on GCE node
W20260928 08:57:31.256982 10448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:31.257154 10447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:31.256987 10450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:31.257431 10442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:31.257681 10442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:31.258831 10442 hybrid_clock.cc:648] HybridClock initialized: now 1790585851258822 us; error 26 us; skew 500 ppm
I20260928 08:57:31.260063 10442 webserver.cc:460] Webserver started at http://127.23.167.193:40891/ using document root <none> and password file <none>
I20260928 08:57:31.260273 10442 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:31.260317 10442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:31.260440 10442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:31.261539 10442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-0/data/instance:
uuid: "fd700b57d5144edaaa12a1fd4cf8be68"
format_stamp: "Formatted at 2026-09-28 08:57:31 on dist-test-slave-pbjp"
server_key: "9196cce6eead636722bda32998219181"
server_key_iv: "22ce8f52e220e186bb347d302d912a72"
server_key_version: "encryptionkey@0"
I20260928 08:57:31.261857 10442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance:
uuid: "fd700b57d5144edaaa12a1fd4cf8be68"
format_stamp: "Formatted at 2026-09-28 08:57:31 on dist-test-slave-pbjp"
server_key: "9196cce6eead636722bda32998219181"
server_key_iv: "22ce8f52e220e186bb347d302d912a72"
server_key_version: "encryptionkey@0"
I20260928 08:57:31.263057 10442 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:57:31.263736 10456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:31.263875 10442 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:31.263969 10442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "fd700b57d5144edaaa12a1fd4cf8be68"
format_stamp: "Formatted at 2026-09-28 08:57:31 on dist-test-slave-pbjp"
server_key: "9196cce6eead636722bda32998219181"
server_key_iv: "22ce8f52e220e186bb347d302d912a72"
server_key_version: "encryptionkey@0"
I20260928 08:57:31.264029 10442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:31.273818 10442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:31.274106 10442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:31.274225 10442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:31.274422 10442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:57:31.274708 10442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:57:31.274739 10442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:31.274768 10442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:57:31.274791 10442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:31.280294 10442 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:41919
I20260928 08:57:31.280335 10569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:41919 every 8 connection(s)
I20260928 08:57:31.280653 10442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
I20260928 08:57:31.285531 10422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:31.281439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:54075 (local address 127.23.167.254:46193)
0928 08:57:31.281490 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:31.281493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:31.281728 (+ 235us) server_negotiation.cc:407] Connection header received
0928 08:57:31.282212 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:31.282213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:31.282227 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:31.282276 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:31.282901 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.283394 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.284315 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.284495 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.284541 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:31.285310 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:31.285310 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:31.285311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:31.285314 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:31.285326 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:31.285329 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:31.285365 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:31.285434 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 08:57:31.285454 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 08:57:31.285738 10572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:31.281421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:46193 (local address 127.23.167.193:54075)
0928 08:57:31.281612 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:31.281648 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:31.282127 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:31.282339 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:31.282342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:31.282350 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:57:31.282837 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:31.282842 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.283466 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.283467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:31.284253 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:31.284255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.284559 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.284560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:31.284664 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:31.285193 (+ 529us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:57:31.285194 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:57:31.285223 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:57:31.285245 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:57:31.285379 (+ 134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:57:31.285381 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:57:31.285384 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:57:31.285429 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 08:57:31.285508 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
I20260928 08:57:31.286201 10570 heartbeater.cc:347] Connected to a master server at 127.23.167.254:46193
I20260928 08:57:31.286284 10570 heartbeater.cc:464] Registering TS with master...
I20260928 08:57:31.286427 10570 heartbeater.cc:511] Master 127.23.167.254:46193 requested a full tablet report, sending...
I20260928 08:57:31.286895 10382 ts_manager.cc:194] Registered new tserver with Master: fd700b57d5144edaaa12a1fd4cf8be68 (127.23.167.193:41919)
I20260928 08:57:31.287513 10382 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.167.193:54075
I20260928 08:57:31.287894 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 10442
I20260928 08:57:31.287983 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance
I20260928 08:57:31.288174 24223 external_mini_cluster.cc:1428] Setting key bbbce6ccc487494d08978903b20bbbab
I20260928 08:57:31.289134 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:0
--local_ip_for_outbound_sockets=127.23.167.194
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:46193
--builtin_ntp_servers=127.23.167.212:45157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 08:57:31.370923 10573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:31.371151 10573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:31.371181 10573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:31.372406 10573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:31.372455 10573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:31.372491 10573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:57:31.373972 10573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:45157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:46193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:31.374220 10573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:31.374516 10573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:31.379299 10579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:31.379311 10578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:31.379303 10581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:31.379534 10573 server_base.cc:1034] running on GCE node
I20260928 08:57:31.379776 10573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:31.380025 10573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:31.381172 10573 hybrid_clock.cc:648] HybridClock initialized: now 1790585851381155 us; error 28 us; skew 500 ppm
I20260928 08:57:31.382381 10573 webserver.cc:460] Webserver started at http://127.23.167.194:39243/ using document root <none> and password file <none>
I20260928 08:57:31.382615 10573 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:31.382664 10573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:31.382793 10573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:31.383888 10573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-1/data/instance:
uuid: "97fe6d9263cd496b9210258a2de6d13a"
format_stamp: "Formatted at 2026-09-28 08:57:31 on dist-test-slave-pbjp"
server_key: "d5a1437107faea71ba4c830b15d5eca2"
server_key_iv: "8ecfe7dc995cac1ac3338680ad85e59a"
server_key_version: "encryptionkey@0"
I20260928 08:57:31.384217 10573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance:
uuid: "97fe6d9263cd496b9210258a2de6d13a"
format_stamp: "Formatted at 2026-09-28 08:57:31 on dist-test-slave-pbjp"
server_key: "d5a1437107faea71ba4c830b15d5eca2"
server_key_iv: "8ecfe7dc995cac1ac3338680ad85e59a"
server_key_version: "encryptionkey@0"
I20260928 08:57:31.385465 10573 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:57:31.386181 10587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:31.386335 10573 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:57:31.386425 10573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "97fe6d9263cd496b9210258a2de6d13a"
format_stamp: "Formatted at 2026-09-28 08:57:31 on dist-test-slave-pbjp"
server_key: "d5a1437107faea71ba4c830b15d5eca2"
server_key_iv: "8ecfe7dc995cac1ac3338680ad85e59a"
server_key_version: "encryptionkey@0"
I20260928 08:57:31.386492 10573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:31.396763 10573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:31.397027 10573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:31.397167 10573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:31.397356 10573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:57:31.397631 10573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:57:31.397662 10573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:31.397689 10573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:57:31.397709 10573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:31.403303 10573 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:33713
I20260928 08:57:31.403345 10700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:33713 every 8 connection(s)
I20260928 08:57:31.403699 10573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
I20260928 08:57:31.408834 10422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:31.404462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:47121 (local address 127.23.167.254:46193)
0928 08:57:31.404517 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:31.404519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:31.404734 (+ 215us) server_negotiation.cc:407] Connection header received
0928 08:57:31.405313 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:31.405314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:31.405327 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:31.405380 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:31.406049 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.406566 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.407538 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.407724 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.407780 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:31.408576 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:31.408577 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:31.408578 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:31.408582 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:31.408600 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:31.408603 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:31.408644 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:31.408719 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 08:57:31.408747 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 08:57:31.409102 10703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:31.404438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:46193 (local address 127.23.167.194:47121)
0928 08:57:31.404616 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:31.404650 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:31.405209 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:31.405453 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:31.405457 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:31.405468 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:57:31.405960 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:31.405964 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.406665 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.406666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:31.407452 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:31.407454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.407801 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.407802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:31.407927 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:31.408444 (+ 517us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:57:31.408444 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:57:31.408474 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:57:31.408497 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:57:31.408650 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:57:31.408655 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:57:31.408658 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:57:31.408718 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 08:57:31.408826 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
I20260928 08:57:31.409541 10701 heartbeater.cc:347] Connected to a master server at 127.23.167.254:46193
I20260928 08:57:31.409626 10701 heartbeater.cc:464] Registering TS with master...
I20260928 08:57:31.409796 10701 heartbeater.cc:511] Master 127.23.167.254:46193 requested a full tablet report, sending...
I20260928 08:57:31.410122 10382 ts_manager.cc:194] Registered new tserver with Master: 97fe6d9263cd496b9210258a2de6d13a (127.23.167.194:33713)
I20260928 08:57:31.410501 10382 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.167.194:47121
I20260928 08:57:31.413925 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 10573
I20260928 08:57:31.414016 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance
I20260928 08:57:31.414209 24223 external_mini_cluster.cc:1428] Setting key ff8b695b2dd0c05b9066a9213fffc688
I20260928 08:57:31.415113 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:0
--local_ip_for_outbound_sockets=127.23.167.195
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:46193
--builtin_ntp_servers=127.23.167.212:45157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 08:57:31.493898 10704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:31.494107 10704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:31.494135 10704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:31.495285 10704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:31.495317 10704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:31.495349 10704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:57:31.496655 10704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:45157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:46193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:31.496874 10704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:31.497169 10704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:31.501806 10709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:31.501933 10710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:31.501935 10712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:31.502341 10704 server_base.cc:1034] running on GCE node
I20260928 08:57:31.502535 10704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:31.502750 10704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:31.503887 10704 hybrid_clock.cc:648] HybridClock initialized: now 1790585851503872 us; error 26 us; skew 500 ppm
I20260928 08:57:31.508579 10704 webserver.cc:460] Webserver started at http://127.23.167.195:43375/ using document root <none> and password file <none>
I20260928 08:57:31.508777 10704 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:31.508817 10704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:31.508903 10704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:31.510005 10704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-2/data/instance:
uuid: "0dbc7ab0d35c4639b8df28e3b8a2ee06"
format_stamp: "Formatted at 2026-09-28 08:57:31 on dist-test-slave-pbjp"
server_key: "9adc4de546f62203a66865e6b33dbd37"
server_key_iv: "d47ad514d9b3060cee537cbb0f2cfbb5"
server_key_version: "encryptionkey@0"
I20260928 08:57:31.510288 10704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance:
uuid: "0dbc7ab0d35c4639b8df28e3b8a2ee06"
format_stamp: "Formatted at 2026-09-28 08:57:31 on dist-test-slave-pbjp"
server_key: "9adc4de546f62203a66865e6b33dbd37"
server_key_iv: "d47ad514d9b3060cee537cbb0f2cfbb5"
server_key_version: "encryptionkey@0"
I20260928 08:57:31.511483 10704 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:57:31.512118 10718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:31.512264 10704 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:57:31.512342 10704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "0dbc7ab0d35c4639b8df28e3b8a2ee06"
format_stamp: "Formatted at 2026-09-28 08:57:31 on dist-test-slave-pbjp"
server_key: "9adc4de546f62203a66865e6b33dbd37"
server_key_iv: "d47ad514d9b3060cee537cbb0f2cfbb5"
server_key_version: "encryptionkey@0"
I20260928 08:57:31.512400 10704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:31.523118 10704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:31.523401 10704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:31.523530 10704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:31.523730 10704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:57:31.524055 10704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:57:31.524089 10704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:31.524119 10704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:57:31.524137 10704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:31.529578 10704 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:32929
I20260928 08:57:31.529644 10831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:32929 every 8 connection(s)
I20260928 08:57:31.529951 10704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
I20260928 08:57:31.534976 10422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:31.530657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:45615 (local address 127.23.167.254:46193)
0928 08:57:31.530710 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:31.530712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:31.530932 (+ 220us) server_negotiation.cc:407] Connection header received
0928 08:57:31.531444 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:31.531445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:31.531457 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:31.531546 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:31.532204 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.532699 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.533705 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.533901 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.533955 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:31.534695 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:31.534697 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:31.534697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:31.534702 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:31.534718 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:31.534722 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:31.534765 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:31.534870 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 08:57:31.534897 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 08:57:31.535151 10834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:31.530639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:46193 (local address 127.23.167.195:45615)
0928 08:57:31.530822 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:31.530856 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:31.531350 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:31.531620 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:31.531624 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:31.531635 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:57:31.532117 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:31.532122 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.532782 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.532783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:31.533627 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:31.533629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.533968 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.533969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:31.534095 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:31.534566 (+ 471us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:57:31.534566 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:57:31.534595 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:57:31.534617 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:57:31.534770 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:57:31.534775 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:57:31.534778 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:57:31.534837 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 08:57:31.534941 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
I20260928 08:57:31.535665 10832 heartbeater.cc:347] Connected to a master server at 127.23.167.254:46193
I20260928 08:57:31.535749 10832 heartbeater.cc:464] Registering TS with master...
I20260928 08:57:31.535920 10832 heartbeater.cc:511] Master 127.23.167.254:46193 requested a full tablet report, sending...
I20260928 08:57:31.536278 10382 ts_manager.cc:194] Registered new tserver with Master: 0dbc7ab0d35c4639b8df28e3b8a2ee06 (127.23.167.195:32929)
I20260928 08:57:31.536631 10382 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.167.195:45615
I20260928 08:57:31.539875 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 10704
I20260928 08:57:31.539953 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance
I20260928 08:57:31.540150 24223 external_mini_cluster.cc:1428] Setting key b0f667cf6cdc08298c424fcc9917971d
I20260928 08:57:31.540941 24223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:57:31.544806 10422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:31.542233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59276 (local address 127.23.167.254:46193)
0928 08:57:31.542285 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:31.542287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:31.542384 (+ 97us) server_negotiation.cc:407] Connection header received
0928 08:57:31.542421 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:31.542422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:31.542430 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:31.542473 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:31.542626 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.543073 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.543769 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.543941 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.543982 (+ 41us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:31.544591 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:31.544592 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:31.544592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:31.544596 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:31.544606 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:31.544609 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:31.544658 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:31.544722 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 08:57:31.544730 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 08:57:31.546490 10382 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:57:31.547981 10382 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:57:31.553922 10844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:31.550227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43408 (local address 127.23.167.193:41919)
0928 08:57:31.550402 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:31.550407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:31.550530 (+ 123us) server_negotiation.cc:407] Connection header received
0928 08:57:31.550579 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:31.550581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:31.550596 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:31.550666 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:57:31.550828 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.551625 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.552986 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.553558 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.553622 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:31.553710 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:31.553798 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 08:57:31.553835 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":89,"threads_started":1}
I20260928 08:57:31.554066 10841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:31.549928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:33713 (local address 127.0.0.1:57642)
0928 08:57:31.550101 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:31.550143 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:31.550206 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:31.550627 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:31.550630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:31.550632 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:57:31.550736 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:31.550738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.552085 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.552086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:31.552846 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:31.552848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.553743 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.553744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:31.553804 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:31.553808 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:57:31.553837 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 08:57:31.553876 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":27,"thread_start_us":68,"threads_started":1}
I20260928 08:57:31.554217 10845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:31.550281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57642 (local address 127.23.167.194:33713)
0928 08:57:31.550443 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:31.550446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:31.550455 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:57:31.550488 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:31.550498 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:31.550512 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:31.550614 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:57:31.550860 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.551988 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.552926 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.553645 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.553977 (+ 332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:31.554081 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:31.554104 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 08:57:31.554138 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":90,"threads_started":1}
I20260928 08:57:31.554302 10840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:31.549935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:32929 (local address 127.0.0.1:43346)
0928 08:57:31.550222 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:31.550233 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:31.550286 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:31.550630 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:31.550632 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:31.550634 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:57:31.550729 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:31.550731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.552034 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.552035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:31.553122 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:31.553124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.553947 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.553948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:31.553999 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:31.554003 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:57:31.554029 (+ 26us) client_negotiation.cc:240] Negotiation successful
0928 08:57:31.554062 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":30,"thread_start_us":106,"threads_started":1}
I20260928 08:57:31.554307 10842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:31.550062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43346 (local address 127.23.167.195:32929)
0928 08:57:31.550431 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:31.550435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:31.550446 (+ 11us) server_negotiation.cc:407] Connection header received
0928 08:57:31.550486 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:31.550488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:31.550503 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:31.550797 (+ 294us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:57:31.550855 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.551950 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.553195 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.553863 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.554103 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:31.554182 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:31.554202 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 08:57:31.554237 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":71,"threads_started":1}
I20260928 08:57:31.553939 10843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:31.550135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:41919 (local address 127.0.0.1:43408)
0928 08:57:31.550461 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:31.550470 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:31.550523 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:31.550729 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:31.550730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:31.550732 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:57:31.550759 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:31.550760 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.551701 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.551702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:31.552889 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:31.552892 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.553631 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.553632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:31.553704 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:31.553717 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 08:57:31.553783 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 08:57:31.553835 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":1,"thread_start_us":75,"threads_started":1}
I20260928 08:57:31.554929 10635 tablet_service.cc:1467] Processing CreateTablet for tablet 8fdbfbe2c2c147dab22de949f9d6ef49 (DEFAULT_TABLE table=test-table [id=952b57e3bdef4858b079aff29e2baa4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:57:31.555071 10766 tablet_service.cc:1467] Processing CreateTablet for tablet 8fdbfbe2c2c147dab22de949f9d6ef49 (DEFAULT_TABLE table=test-table [id=952b57e3bdef4858b079aff29e2baa4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:57:31.555058 10504 tablet_service.cc:1467] Processing CreateTablet for tablet 8fdbfbe2c2c147dab22de949f9d6ef49 (DEFAULT_TABLE table=test-table [id=952b57e3bdef4858b079aff29e2baa4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:57:31.555236 10635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fdbfbe2c2c147dab22de949f9d6ef49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:31.555395 10504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fdbfbe2c2c147dab22de949f9d6ef49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:31.555394 10766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fdbfbe2c2c147dab22de949f9d6ef49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:31.557852 10846 tablet_bootstrap.cc:492] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 97fe6d9263cd496b9210258a2de6d13a: Bootstrap starting.
I20260928 08:57:31.558454 10846 tablet_bootstrap.cc:654] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 97fe6d9263cd496b9210258a2de6d13a: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:31.558670 10848 tablet_bootstrap.cc:492] T 8fdbfbe2c2c147dab22de949f9d6ef49 P fd700b57d5144edaaa12a1fd4cf8be68: Bootstrap starting.
I20260928 08:57:31.558770 10846 log.cc:824] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 97fe6d9263cd496b9210258a2de6d13a: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:31.559264 10848 tablet_bootstrap.cc:654] T 8fdbfbe2c2c147dab22de949f9d6ef49 P fd700b57d5144edaaa12a1fd4cf8be68: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:31.559481 10846 tablet_bootstrap.cc:492] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 97fe6d9263cd496b9210258a2de6d13a: No bootstrap required, opened a new log
I20260928 08:57:31.559549 10846 ts_tablet_manager.cc:1397] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 97fe6d9263cd496b9210258a2de6d13a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:57:31.559576 10848 log.cc:824] T 8fdbfbe2c2c147dab22de949f9d6ef49 P fd700b57d5144edaaa12a1fd4cf8be68: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:31.558485 10847 tablet_bootstrap.cc:492] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 0dbc7ab0d35c4639b8df28e3b8a2ee06: Bootstrap starting.
I20260928 08:57:31.560242 10848 tablet_bootstrap.cc:492] T 8fdbfbe2c2c147dab22de949f9d6ef49 P fd700b57d5144edaaa12a1fd4cf8be68: No bootstrap required, opened a new log
I20260928 08:57:31.560307 10848 ts_tablet_manager.cc:1397] T 8fdbfbe2c2c147dab22de949f9d6ef49 P fd700b57d5144edaaa12a1fd4cf8be68: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:57:31.560387 10847 tablet_bootstrap.cc:654] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 0dbc7ab0d35c4639b8df28e3b8a2ee06: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:31.560704 10847 log.cc:824] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 0dbc7ab0d35c4639b8df28e3b8a2ee06: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:31.561419 10847 tablet_bootstrap.cc:492] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 0dbc7ab0d35c4639b8df28e3b8a2ee06: No bootstrap required, opened a new log
I20260928 08:57:31.561494 10847 ts_tablet_manager.cc:1397] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 0dbc7ab0d35c4639b8df28e3b8a2ee06: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 08:57:31.562192 10848 raft_consensus.cc:348] T 8fdbfbe2c2c147dab22de949f9d6ef49 P fd700b57d5144edaaa12a1fd4cf8be68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97fe6d9263cd496b9210258a2de6d13a" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33713 } } peers { permanent_uuid: "0dbc7ab0d35c4639b8df28e3b8a2ee06" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 32929 } } peers { permanent_uuid: "fd700b57d5144edaaa12a1fd4cf8be68" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 41919 } }
I20260928 08:57:31.562165 10846 raft_consensus.cc:348] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 97fe6d9263cd496b9210258a2de6d13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97fe6d9263cd496b9210258a2de6d13a" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33713 } } peers { permanent_uuid: "0dbc7ab0d35c4639b8df28e3b8a2ee06" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 32929 } } peers { permanent_uuid: "fd700b57d5144edaaa12a1fd4cf8be68" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 41919 } }
I20260928 08:57:31.562321 10848 raft_consensus.cc:374] T 8fdbfbe2c2c147dab22de949f9d6ef49 P fd700b57d5144edaaa12a1fd4cf8be68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:31.562327 10846 raft_consensus.cc:374] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 97fe6d9263cd496b9210258a2de6d13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:31.562352 10848 raft_consensus.cc:729] T 8fdbfbe2c2c147dab22de949f9d6ef49 P fd700b57d5144edaaa12a1fd4cf8be68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd700b57d5144edaaa12a1fd4cf8be68, State: Initialized, Role: FOLLOWER
I20260928 08:57:31.562357 10846 raft_consensus.cc:729] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 97fe6d9263cd496b9210258a2de6d13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97fe6d9263cd496b9210258a2de6d13a, State: Initialized, Role: FOLLOWER
I20260928 08:57:31.562484 10846 consensus_queue.cc:260] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 97fe6d9263cd496b9210258a2de6d13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97fe6d9263cd496b9210258a2de6d13a" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33713 } } peers { permanent_uuid: "0dbc7ab0d35c4639b8df28e3b8a2ee06" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 32929 } } peers { permanent_uuid: "fd700b57d5144edaaa12a1fd4cf8be68" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 41919 } }
I20260928 08:57:31.562479 10848 consensus_queue.cc:260] T 8fdbfbe2c2c147dab22de949f9d6ef49 P fd700b57d5144edaaa12a1fd4cf8be68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97fe6d9263cd496b9210258a2de6d13a" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33713 } } peers { permanent_uuid: "0dbc7ab0d35c4639b8df28e3b8a2ee06" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 32929 } } peers { permanent_uuid: "fd700b57d5144edaaa12a1fd4cf8be68" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 41919 } }
I20260928 08:57:31.562726 10848 ts_tablet_manager.cc:1428] T 8fdbfbe2c2c147dab22de949f9d6ef49 P fd700b57d5144edaaa12a1fd4cf8be68: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 08:57:31.562732 10846 ts_tablet_manager.cc:1428] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 97fe6d9263cd496b9210258a2de6d13a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:57:31.562836 10701 heartbeater.cc:503] Master 127.23.167.254:46193 was elected leader, sending a full tablet report...
I20260928 08:57:31.562842 10570 heartbeater.cc:503] Master 127.23.167.254:46193 was elected leader, sending a full tablet report...
I20260928 08:57:31.563671 10847 raft_consensus.cc:348] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 0dbc7ab0d35c4639b8df28e3b8a2ee06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97fe6d9263cd496b9210258a2de6d13a" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33713 } } peers { permanent_uuid: "0dbc7ab0d35c4639b8df28e3b8a2ee06" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 32929 } } peers { permanent_uuid: "fd700b57d5144edaaa12a1fd4cf8be68" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 41919 } }
I20260928 08:57:31.563819 10847 raft_consensus.cc:374] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 0dbc7ab0d35c4639b8df28e3b8a2ee06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:31.563858 10847 raft_consensus.cc:729] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 0dbc7ab0d35c4639b8df28e3b8a2ee06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dbc7ab0d35c4639b8df28e3b8a2ee06, State: Initialized, Role: FOLLOWER
I20260928 08:57:31.563952 10847 consensus_queue.cc:260] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 0dbc7ab0d35c4639b8df28e3b8a2ee06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97fe6d9263cd496b9210258a2de6d13a" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33713 } } peers { permanent_uuid: "0dbc7ab0d35c4639b8df28e3b8a2ee06" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 32929 } } peers { permanent_uuid: "fd700b57d5144edaaa12a1fd4cf8be68" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 41919 } }
I20260928 08:57:31.564170 10847 ts_tablet_manager.cc:1428] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 0dbc7ab0d35c4639b8df28e3b8a2ee06: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:57:31.564239 10832 heartbeater.cc:503] Master 127.23.167.254:46193 was elected leader, sending a full tablet report...
W20260928 08:57:31.654294 10702 tablet.cc:2369] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 97fe6d9263cd496b9210258a2de6d13a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:57:31.694644 10852 raft_consensus.cc:482] T 8fdbfbe2c2c147dab22de949f9d6ef49 P fd700b57d5144edaaa12a1fd4cf8be68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:57:31.694770 10852 raft_consensus.cc:504] T 8fdbfbe2c2c147dab22de949f9d6ef49 P fd700b57d5144edaaa12a1fd4cf8be68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97fe6d9263cd496b9210258a2de6d13a" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33713 } } peers { permanent_uuid: "0dbc7ab0d35c4639b8df28e3b8a2ee06" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 32929 } } peers { permanent_uuid: "fd700b57d5144edaaa12a1fd4cf8be68" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 41919 } }
I20260928 08:57:31.695101 10852 leader_election.cc:288] T 8fdbfbe2c2c147dab22de949f9d6ef49 P fd700b57d5144edaaa12a1fd4cf8be68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97fe6d9263cd496b9210258a2de6d13a (127.23.167.194:33713), 0dbc7ab0d35c4639b8df28e3b8a2ee06 (127.23.167.195:32929)
I20260928 08:57:31.698037 10845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:31.695358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:40111 (local address 127.23.167.194:33713)
0928 08:57:31.695441 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:31.695443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:31.695453 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:57:31.695482 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:31.695483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:31.695490 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:31.695544 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:57:31.695690 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.696176 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.697294 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.697749 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.697799 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:31.697806 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:31.697959 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 08:57:31.697984 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 08:57:31.698072 10572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:31.695273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:33713 (local address 127.23.167.193:40111)
0928 08:57:31.695329 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:31.695341 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:31.695386 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:31.695562 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:31.695563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:31.695565 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:57:31.695631 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:31.695633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.696254 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.696255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:31.697221 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:31.697224 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.697825 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.697826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:31.697895 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:31.697898 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:57:31.697955 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 08:57:31.697995 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 08:57:31.698375 10842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:31.695360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:47443 (local address 127.23.167.195:32929)
0928 08:57:31.695425 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:31.695426 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:31.695640 (+ 214us) server_negotiation.cc:407] Connection header received
0928 08:57:31.695686 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:31.695687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:31.695696 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:31.695743 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:57:31.695872 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.696524 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.697587 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.698018 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.698076 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:31.698083 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:31.698271 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 08:57:31.698310 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 08:57:31.698376 10855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:31.695280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:32929 (local address 127.23.167.193:47443)
0928 08:57:31.695579 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:31.695587 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:31.695638 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:31.695749 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:31.695750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:31.695751 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:57:31.695820 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:31.695822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.696603 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.696603 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:31.697505 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:31.697507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.698097 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.698098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:31.698169 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:31.698173 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:57:31.698257 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 08:57:31.698296 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":153,"thread_start_us":83,"threads_started":1}
I20260928 08:57:31.698457 10655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fdbfbe2c2c147dab22de949f9d6ef49" candidate_uuid: "fd700b57d5144edaaa12a1fd4cf8be68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97fe6d9263cd496b9210258a2de6d13a" is_pre_election: true
I20260928 08:57:31.698611 10655 raft_consensus.cc:2454] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 97fe6d9263cd496b9210258a2de6d13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd700b57d5144edaaa12a1fd4cf8be68 in term 0.
I20260928 08:57:31.698714 10786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fdbfbe2c2c147dab22de949f9d6ef49" candidate_uuid: "fd700b57d5144edaaa12a1fd4cf8be68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dbc7ab0d35c4639b8df28e3b8a2ee06" is_pre_election: true
I20260928 08:57:31.698850 10786 raft_consensus.cc:2454] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 0dbc7ab0d35c4639b8df28e3b8a2ee06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd700b57d5144edaaa12a1fd4cf8be68 in term 0.
I20260928 08:57:31.698848 10458 leader_election.cc:302] T 8fdbfbe2c2c147dab22de949f9d6ef49 P fd700b57d5144edaaa12a1fd4cf8be68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97fe6d9263cd496b9210258a2de6d13a, fd700b57d5144edaaa12a1fd4cf8be68; no voters:
I20260928 08:57:31.698964 10852 raft_consensus.cc:2781] T 8fdbfbe2c2c147dab22de949f9d6ef49 P fd700b57d5144edaaa12a1fd4cf8be68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:57:31.699044 10852 raft_consensus.cc:482] T 8fdbfbe2c2c147dab22de949f9d6ef49 P fd700b57d5144edaaa12a1fd4cf8be68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:57:31.699070 10852 raft_consensus.cc:3037] T 8fdbfbe2c2c147dab22de949f9d6ef49 P fd700b57d5144edaaa12a1fd4cf8be68 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:31.699772 10852 raft_consensus.cc:504] T 8fdbfbe2c2c147dab22de949f9d6ef49 P fd700b57d5144edaaa12a1fd4cf8be68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97fe6d9263cd496b9210258a2de6d13a" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33713 } } peers { permanent_uuid: "0dbc7ab0d35c4639b8df28e3b8a2ee06" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 32929 } } peers { permanent_uuid: "fd700b57d5144edaaa12a1fd4cf8be68" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 41919 } }
I20260928 08:57:31.699889 10852 leader_election.cc:288] T 8fdbfbe2c2c147dab22de949f9d6ef49 P fd700b57d5144edaaa12a1fd4cf8be68 [CANDIDATE]: Term 1 election: Requested vote from peers 97fe6d9263cd496b9210258a2de6d13a (127.23.167.194:33713), 0dbc7ab0d35c4639b8df28e3b8a2ee06 (127.23.167.195:32929)
I20260928 08:57:31.700033 10786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fdbfbe2c2c147dab22de949f9d6ef49" candidate_uuid: "fd700b57d5144edaaa12a1fd4cf8be68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dbc7ab0d35c4639b8df28e3b8a2ee06"
I20260928 08:57:31.700098 10786 raft_consensus.cc:3037] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 0dbc7ab0d35c4639b8df28e3b8a2ee06 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:31.700839 10786 raft_consensus.cc:2454] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 0dbc7ab0d35c4639b8df28e3b8a2ee06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd700b57d5144edaaa12a1fd4cf8be68 in term 1.
I20260928 08:57:31.700939 10655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fdbfbe2c2c147dab22de949f9d6ef49" candidate_uuid: "fd700b57d5144edaaa12a1fd4cf8be68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97fe6d9263cd496b9210258a2de6d13a"
I20260928 08:57:31.701025 10460 leader_election.cc:302] T 8fdbfbe2c2c147dab22de949f9d6ef49 P fd700b57d5144edaaa12a1fd4cf8be68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dbc7ab0d35c4639b8df28e3b8a2ee06, fd700b57d5144edaaa12a1fd4cf8be68; no voters:
I20260928 08:57:31.701017 10655 raft_consensus.cc:3037] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 97fe6d9263cd496b9210258a2de6d13a [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:31.701124 10852 raft_consensus.cc:2781] T 8fdbfbe2c2c147dab22de949f9d6ef49 P fd700b57d5144edaaa12a1fd4cf8be68 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:57:31.701270 10852 raft_consensus.cc:686] T 8fdbfbe2c2c147dab22de949f9d6ef49 P fd700b57d5144edaaa12a1fd4cf8be68 [term 1 LEADER]: Becoming Leader. State: Replica: fd700b57d5144edaaa12a1fd4cf8be68, State: Running, Role: LEADER
I20260928 08:57:31.701359 10852 consensus_queue.cc:237] T 8fdbfbe2c2c147dab22de949f9d6ef49 P fd700b57d5144edaaa12a1fd4cf8be68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97fe6d9263cd496b9210258a2de6d13a" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33713 } } peers { permanent_uuid: "0dbc7ab0d35c4639b8df28e3b8a2ee06" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 32929 } } peers { permanent_uuid: "fd700b57d5144edaaa12a1fd4cf8be68" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 41919 } }
I20260928 08:57:31.701833 10655 raft_consensus.cc:2454] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 97fe6d9263cd496b9210258a2de6d13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd700b57d5144edaaa12a1fd4cf8be68 in term 1.
I20260928 08:57:31.702174 10381 catalog_manager.cc:5495] T 8fdbfbe2c2c147dab22de949f9d6ef49 P fd700b57d5144edaaa12a1fd4cf8be68 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd700b57d5144edaaa12a1fd4cf8be68 (127.23.167.193). New cstate: current_term: 1 leader_uuid: "fd700b57d5144edaaa12a1fd4cf8be68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97fe6d9263cd496b9210258a2de6d13a" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dbc7ab0d35c4639b8df28e3b8a2ee06" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 32929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd700b57d5144edaaa12a1fd4cf8be68" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 41919 } health_report { overall_health: HEALTHY } } }
I20260928 08:57:31.737270 10844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:31.734836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43418 (local address 127.23.167.193:41919)
0928 08:57:31.734897 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:31.734899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:31.734907 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:57:31.734946 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:31.734947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:31.734970 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:31.735032 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:57:31.735152 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.735722 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.736466 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:31.736660 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:31.736714 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:31.736865 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:57:31.736959 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:57:31.737047 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:31.737152 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 08:57:31.737164 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 08:57:31.739205 10655 raft_consensus.cc:1261] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 97fe6d9263cd496b9210258a2de6d13a [term 1 FOLLOWER]: Refusing update from remote peer fd700b57d5144edaaa12a1fd4cf8be68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:57:31.739205 10786 raft_consensus.cc:1261] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 0dbc7ab0d35c4639b8df28e3b8a2ee06 [term 1 FOLLOWER]: Refusing update from remote peer fd700b57d5144edaaa12a1fd4cf8be68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:57:31.739604 10856 consensus_queue.cc:1035] T 8fdbfbe2c2c147dab22de949f9d6ef49 P fd700b57d5144edaaa12a1fd4cf8be68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97fe6d9263cd496b9210258a2de6d13a" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:57:31.739737 10852 consensus_queue.cc:1035] T 8fdbfbe2c2c147dab22de949f9d6ef49 P fd700b57d5144edaaa12a1fd4cf8be68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dbc7ab0d35c4639b8df28e3b8a2ee06" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 32929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:57:31.742601 10860 mvcc.cc:205] Tried to move back new op lower bound from 7334239648720736256 to 7334239648569331712. Current Snapshot: MvccSnapshot[applied={T|T < 7334239648720736256}]
I20260928 08:57:31.742964 10861 mvcc.cc:205] Tried to move back new op lower bound from 7334239648720736256 to 7334239648569331712. Current Snapshot: MvccSnapshot[applied={T|T < 7334239648720736256}]
I20260928 08:57:31.744441 10382 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:57:31.744529 10382 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:57:31.745250 10382 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 545dc129a06c43449a55fa4234ee710f: Sending DeleteTablet for 3 replicas of tablet 8fdbfbe2c2c147dab22de949f9d6ef49
I20260928 08:57:31.745606 10766 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fdbfbe2c2c147dab22de949f9d6ef49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:57:31 UTC) from {username='slave'} at 127.0.0.1:43346
I20260928 08:57:31.745659 10504 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fdbfbe2c2c147dab22de949f9d6ef49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:57:31 UTC) from {username='slave'} at 127.0.0.1:43408
I20260928 08:57:31.745641 10635 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fdbfbe2c2c147dab22de949f9d6ef49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:57:31 UTC) from {username='slave'} at 127.0.0.1:57642
I20260928 08:57:31.745826 10869 tablet_replica.cc:330] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 0dbc7ab0d35c4639b8df28e3b8a2ee06: stopping tablet replica
I20260928 08:57:31.745940 10869 raft_consensus.cc:2229] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 0dbc7ab0d35c4639b8df28e3b8a2ee06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:57:31.746057 10869 raft_consensus.cc:2258] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 0dbc7ab0d35c4639b8df28e3b8a2ee06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:57:31.746114 10870 tablet_replica.cc:330] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 97fe6d9263cd496b9210258a2de6d13a: stopping tablet replica
I20260928 08:57:31.746186 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 10442
I20260928 08:57:31.746212 10870 raft_consensus.cc:2229] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 97fe6d9263cd496b9210258a2de6d13a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:57:31.746304 10870 raft_consensus.cc:2258] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 97fe6d9263cd496b9210258a2de6d13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:57:31.747109 10870 ts_tablet_manager.cc:1905] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 97fe6d9263cd496b9210258a2de6d13a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:57:31.748895 10870 ts_tablet_manager.cc:1918] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 97fe6d9263cd496b9210258a2de6d13a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:57:31.748960 10870 log.cc:1196] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 97fe6d9263cd496b9210258a2de6d13a: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-1/wal/wals/8fdbfbe2c2c147dab22de949f9d6ef49
I20260928 08:57:31.749202 10870 ts_tablet_manager.cc:1939] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 97fe6d9263cd496b9210258a2de6d13a: Deleting consensus metadata
I20260928 08:57:31.749661 10367 catalog_manager.cc:4841] TS 97fe6d9263cd496b9210258a2de6d13a (127.23.167.194:33713): tablet 8fdbfbe2c2c147dab22de949f9d6ef49 (table test-table [id=952b57e3bdef4858b079aff29e2baa4d]) successfully deleted
I20260928 08:57:31.750801 10869 ts_tablet_manager.cc:1905] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 0dbc7ab0d35c4639b8df28e3b8a2ee06: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 08:57:31.751605 10367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.167.193:41919 (error 108)
I20260928 08:57:31.751885 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 10573
I20260928 08:57:31.752059 10843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:31.751936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:41919 (local address 127.0.0.1:43428)
0928 08:57:31.751989 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:31.752028 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.193:41919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 08:57:31.752552 10367 catalog_manager.cc:4584] TS fd700b57d5144edaaa12a1fd4cf8be68 (127.23.167.193:41919): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8fdbfbe2c2c147dab22de949f9d6ef49: Network error: Client connection negotiation failed: client connection to 127.23.167.193:41919: connect: Connection refused (error 111)
I20260928 08:57:31.752751 10869 ts_tablet_manager.cc:1918] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 0dbc7ab0d35c4639b8df28e3b8a2ee06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:57:31.752826 10869 log.cc:1196] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 0dbc7ab0d35c4639b8df28e3b8a2ee06: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585589999956-24223-0/minicluster-data/ts-2/wal/wals/8fdbfbe2c2c147dab22de949f9d6ef49
I20260928 08:57:31.753086 10869 ts_tablet_manager.cc:1939] T 8fdbfbe2c2c147dab22de949f9d6ef49 P 0dbc7ab0d35c4639b8df28e3b8a2ee06: Deleting consensus metadata
I20260928 08:57:31.753515 10369 catalog_manager.cc:4841] TS 0dbc7ab0d35c4639b8df28e3b8a2ee06 (127.23.167.195:32929): tablet 8fdbfbe2c2c147dab22de949f9d6ef49 (table test-table [id=952b57e3bdef4858b079aff29e2baa4d]) successfully deleted
I20260928 08:57:31.757100 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 10704
I20260928 08:57:31.761250 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 10351
2026-09-28T08:57:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (789 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:57:31 dist-test-slave-pbjp krb5kdc[10875](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:57:31 dist-test-slave-pbjp krb5kdc[10875](info): set up 2 sockets
Sep 28 08:57:31 dist-test-slave-pbjp krb5kdc[10875](info): commencing operation
krb5kdc: starting...
W20260928 08:57:32.696640 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.917s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:57:32 dist-test-slave-pbjp krb5kdc[10875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:57:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:57:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.167.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
I20260928 08:57:32.807633 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:37607
--webserver_interface=127.23.167.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:37607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:32.886732 10888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:32.886935 10888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:32.886966 10888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:32.888099 10888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:57:32.888139 10888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:32.888157 10888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:32.888175 10888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:57:32.888188 10888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:57:32.889714 10888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:37607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:37607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:32.889966 10888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:32.890167 10888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:32.894681 10894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:32.894701 10893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:32.894677 10896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:32.894896 10888 server_base.cc:1034] running on GCE node
I20260928 08:57:32.895099 10888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:32.895367 10888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:32.896535 10888 hybrid_clock.cc:648] HybridClock initialized: now 1790585852896520 us; error 29 us; skew 500 ppm
Sep 28 08:57:32 dist-test-slave-pbjp krb5kdc[10875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585852, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:32.898887 10888 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:57:32.899314 10888 webserver.cc:460] Webserver started at http://127.23.167.254:34651/ using document root <none> and password file <none>
I20260928 08:57:32.899559 10888 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:32.899606 10888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:32.899727 10888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:32.900736 10888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "41461c9167a24a3c8b725ce8adbce38c"
format_stamp: "Formatted at 2026-09-28 08:57:32 on dist-test-slave-pbjp"
server_key: "a4b4772102972c27ce9c61e93afe7d13"
server_key_iv: "6af1ceef3504bb8cc1092d81c5ee89f4"
server_key_version: "encryptionkey@0"
I20260928 08:57:32.901093 10888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "41461c9167a24a3c8b725ce8adbce38c"
format_stamp: "Formatted at 2026-09-28 08:57:32 on dist-test-slave-pbjp"
server_key: "a4b4772102972c27ce9c61e93afe7d13"
server_key_iv: "6af1ceef3504bb8cc1092d81c5ee89f4"
server_key_version: "encryptionkey@0"
I20260928 08:57:32.902312 10888 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:57:32.902989 10903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:32.903143 10888 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:57:32.903227 10888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "41461c9167a24a3c8b725ce8adbce38c"
format_stamp: "Formatted at 2026-09-28 08:57:32 on dist-test-slave-pbjp"
server_key: "a4b4772102972c27ce9c61e93afe7d13"
server_key_iv: "6af1ceef3504bb8cc1092d81c5ee89f4"
server_key_version: "encryptionkey@0"
I20260928 08:57:32.903295 10888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:32.916657 10888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:32.918740 10888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:32.918877 10888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:32.922683 10888 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:37607
I20260928 08:57:32.922729 10955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:37607 every 8 connection(s)
I20260928 08:57:32.923050 10888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:57:32.923867 10956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:32.926146 10956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41461c9167a24a3c8b725ce8adbce38c: Bootstrap starting.
I20260928 08:57:32.926692 10956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41461c9167a24a3c8b725ce8adbce38c: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:32.926942 10956 log.cc:824] T 00000000000000000000000000000000 P 41461c9167a24a3c8b725ce8adbce38c: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:32.927603 10956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41461c9167a24a3c8b725ce8adbce38c: No bootstrap required, opened a new log
I20260928 08:57:32.929157 10956 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41461c9167a24a3c8b725ce8adbce38c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41461c9167a24a3c8b725ce8adbce38c" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 37607 } }
I20260928 08:57:32.929292 10956 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41461c9167a24a3c8b725ce8adbce38c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:32.929313 10956 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41461c9167a24a3c8b725ce8adbce38c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41461c9167a24a3c8b725ce8adbce38c, State: Initialized, Role: FOLLOWER
I20260928 08:57:32.929450 10956 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41461c9167a24a3c8b725ce8adbce38c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41461c9167a24a3c8b725ce8adbce38c" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 37607 } }
I20260928 08:57:32.929523 10956 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41461c9167a24a3c8b725ce8adbce38c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:57:32.929545 10956 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41461c9167a24a3c8b725ce8adbce38c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:57:32.929591 10956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41461c9167a24a3c8b725ce8adbce38c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:32.930085 10956 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41461c9167a24a3c8b725ce8adbce38c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41461c9167a24a3c8b725ce8adbce38c" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 37607 } }
I20260928 08:57:32.930168 10956 leader_election.cc:302] T 00000000000000000000000000000000 P 41461c9167a24a3c8b725ce8adbce38c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41461c9167a24a3c8b725ce8adbce38c; no voters:
I20260928 08:57:32.930318 10956 leader_election.cc:288] T 00000000000000000000000000000000 P 41461c9167a24a3c8b725ce8adbce38c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:57:32.930371 10959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41461c9167a24a3c8b725ce8adbce38c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:57:32.930553 10959 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41461c9167a24a3c8b725ce8adbce38c [term 1 LEADER]: Becoming Leader. State: Replica: 41461c9167a24a3c8b725ce8adbce38c, State: Running, Role: LEADER
I20260928 08:57:32.930562 10956 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41461c9167a24a3c8b725ce8adbce38c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:57:32.930657 10959 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41461c9167a24a3c8b725ce8adbce38c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41461c9167a24a3c8b725ce8adbce38c" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 37607 } }
I20260928 08:57:32.931205 10961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41461c9167a24a3c8b725ce8adbce38c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41461c9167a24a3c8b725ce8adbce38c. Latest consensus state: current_term: 1 leader_uuid: "41461c9167a24a3c8b725ce8adbce38c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41461c9167a24a3c8b725ce8adbce38c" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 37607 } } }
I20260928 08:57:32.931308 10961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41461c9167a24a3c8b725ce8adbce38c [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:32.931205 10960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41461c9167a24a3c8b725ce8adbce38c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41461c9167a24a3c8b725ce8adbce38c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41461c9167a24a3c8b725ce8adbce38c" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 37607 } } }
I20260928 08:57:32.931434 10960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41461c9167a24a3c8b725ce8adbce38c [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:32.931571 10968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:57:32.931986 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 10888
I20260928 08:57:32.932092 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:57:32.932091 10968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:57:32.932291 24223 external_mini_cluster.cc:1428] Setting key 8e9e5d0b28bd060de4b64bc310d45739
I20260928 08:57:32.933813 10968 catalog_manager.cc:1347] Generated new cluster ID: d84b71f3019a4524aba4871e91e4ccb9
I20260928 08:57:32.933871 10968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 08:57:32 dist-test-slave-pbjp krb5kdc[10875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:57:32.943612 10978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:32.932743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59950 (local address 127.23.167.254:37607)
0928 08:57:32.933175 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:32.933184 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:32.933214 (+ 30us) server_negotiation.cc:407] Connection header received
0928 08:57:32.933332 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:32.933341 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:32.933417 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:32.933559 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:32.934269 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:32.935663 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:32.936678 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:32.937293 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:32.937686 (+ 393us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:32.940127 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:32.940131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:32.940132 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:32.940145 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:32.942044 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:32.942433 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:32.942436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:32.942437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:32.942534 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:32.942714 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:32.942714 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:32.942714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:32.943005 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:32.943114 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:32.943245 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 08:57:32.943380 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":82,"threads_started":1}
I20260928 08:57:32.947304 10968 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:57:32.947857 10968 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:57:32.956645 10968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41461c9167a24a3c8b725ce8adbce38c: Generated new TSK 0
I20260928 08:57:32.956849 10968 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.167.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
Entry for principal kudu/127.23.167.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab.
I20260928 08:57:32.984491 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:0
--local_ip_for_outbound_sockets=127.23.167.193
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:37607
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:33.066180 10982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:33.066401 10982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:33.066424 10982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:33.067603 10982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:33.067654 10982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:33.067679 10982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:57:33.069207 10982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.193.keytab
--principal=kudu/127.23.167.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:37607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:33.069442 10982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:33.069686 10982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:33.074672 10987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:33.074666 10988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:33.074683 10982 server_base.cc:1034] running on GCE node
W20260928 08:57:33.074661 10990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:33.075104 10982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:33.075332 10982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:33.076483 10982 hybrid_clock.cc:648] HybridClock initialized: now 1790585853076470 us; error 30 us; skew 500 ppm
Sep 28 08:57:33 dist-test-slave-pbjp krb5kdc[10875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585853, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:33.079005 10982 init.cc:377] Logged in from keytab as kudu/127.23.167.193@KRBTEST.COM (short username kudu)
I20260928 08:57:33.079452 10982 webserver.cc:460] Webserver started at http://127.23.167.193:45873/ using document root <none> and password file <none>
I20260928 08:57:33.079661 10982 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:33.079707 10982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:33.079830 10982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:33.080894 10982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-0/data/instance:
uuid: "f86ecc27b4ab485897f78169ccb8e6de"
format_stamp: "Formatted at 2026-09-28 08:57:33 on dist-test-slave-pbjp"
server_key: "0433cac17ec42be6fd7d9f1d7d051c90"
server_key_iv: "0c708da152fa586c17b6aae7bd10ad4c"
server_key_version: "encryptionkey@0"
I20260928 08:57:33.081243 10982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance:
uuid: "f86ecc27b4ab485897f78169ccb8e6de"
format_stamp: "Formatted at 2026-09-28 08:57:33 on dist-test-slave-pbjp"
server_key: "0433cac17ec42be6fd7d9f1d7d051c90"
server_key_iv: "0c708da152fa586c17b6aae7bd10ad4c"
server_key_version: "encryptionkey@0"
I20260928 08:57:33.082445 10982 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:57:33.083196 10997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:33.083360 10982 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:57:33.083443 10982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "f86ecc27b4ab485897f78169ccb8e6de"
format_stamp: "Formatted at 2026-09-28 08:57:33 on dist-test-slave-pbjp"
server_key: "0433cac17ec42be6fd7d9f1d7d051c90"
server_key_iv: "0c708da152fa586c17b6aae7bd10ad4c"
server_key_version: "encryptionkey@0"
I20260928 08:57:33.083534 10982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:33.096210 10982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:33.098198 10982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:33.098325 10982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:33.098551 10982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:57:33.098851 10982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:57:33.098883 10982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:33.098910 10982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:57:33.098934 10982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:33.104588 10982 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:43603
I20260928 08:57:33.104632 11110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:43603 every 8 connection(s)
I20260928 08:57:33.104955 10982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
I20260928 08:57:33.109270 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 10982
I20260928 08:57:33.109371 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance
I20260928 08:57:33.109576 24223 external_mini_cluster.cc:1428] Setting key 2e19e0eb54ee01ccd757b537572f36ba
Sep 28 08:57:33 dist-test-slave-pbjp krb5kdc[10875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585853, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.193@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:57:33.114645 10978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:33.105680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:36441 (local address 127.23.167.254:37607)
0928 08:57:33.105739 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:33.105741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:33.105950 (+ 209us) server_negotiation.cc:407] Connection header received
0928 08:57:33.106481 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:33.106482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:33.106501 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:33.106560 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:33.107585 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.108098 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.109123 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.109342 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.109408 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:33.111761 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:33.111763 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:33.111763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:33.111772 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:33.113245 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:33.113639 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:33.113640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:33.113641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:33.113706 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:33.113954 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:33.113955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:33.113956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:33.114108 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:33.114172 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:33.114422 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 08:57:33.114537 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 08:57:33.114686 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:33.105657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:37607 (local address 127.23.167.193:36441)
0928 08:57:33.105815 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:33.105864 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:33.106382 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:33.106647 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:33.106651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:33.107274 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:57:33.107476 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:33.107481 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.108186 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.108187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:33.108999 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:33.109001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.109497 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.109499 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:33.109659 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:33.110209 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:57:33.110219 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:57:33.111669 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:57:33.113318 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:33.113326 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:33.113327 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:33.113568 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:33.113777 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:33.113778 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:33.113779 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:33.113896 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:33.114168 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:57:33.114171 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:57:33.114299 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 08:57:33.114377 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 08:57:33.114514 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20260928 08:57:33.115281 11111 heartbeater.cc:347] Connected to a master server at 127.23.167.254:37607
I20260928 08:57:33.115401 11111 heartbeater.cc:464] Registering TS with master...
I20260928 08:57:33.115618 11111 heartbeater.cc:511] Master 127.23.167.254:37607 requested a full tablet report, sending...
I20260928 08:57:33.116212 10920 ts_manager.cc:194] Registered new tserver with Master: f86ecc27b4ab485897f78169ccb8e6de (127.23.167.193:43603)
I20260928 08:57:33.116905 10920 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.193@KRBTEST.COM'} at 127.23.167.193:36441
WARNING: no policy specified for kudu/127.23.167.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
Entry for principal kudu/127.23.167.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab.
I20260928 08:57:33.135190 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:0
--local_ip_for_outbound_sockets=127.23.167.194
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:37607
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:33.214953 11116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:33.215173 11116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:33.215204 11116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:33.216390 11116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:33.216434 11116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:33.216468 11116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:57:33.217978 11116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.194.keytab
--principal=kudu/127.23.167.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:37607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:33.218225 11116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:33.218436 11116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:33.223196 11122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:33.223208 11124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:33.223184 11121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:33.223369 11116 server_base.cc:1034] running on GCE node
I20260928 08:57:33.223662 11116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:33.223865 11116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:33.224997 11116 hybrid_clock.cc:648] HybridClock initialized: now 1790585853224980 us; error 28 us; skew 500 ppm
Sep 28 08:57:33 dist-test-slave-pbjp krb5kdc[10875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585853, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:33.227399 11116 init.cc:377] Logged in from keytab as kudu/127.23.167.194@KRBTEST.COM (short username kudu)
I20260928 08:57:33.227828 11116 webserver.cc:460] Webserver started at http://127.23.167.194:45917/ using document root <none> and password file <none>
I20260928 08:57:33.228008 11116 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:33.228049 11116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:33.228140 11116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:33.229197 11116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-1/data/instance:
uuid: "dbd8629fa10e4963b308740f68d41ea5"
format_stamp: "Formatted at 2026-09-28 08:57:33 on dist-test-slave-pbjp"
server_key: "f81cfd08da2354e019e5402a4427be47"
server_key_iv: "ab9bbac788ca0204c48f1e9c7568f133"
server_key_version: "encryptionkey@0"
I20260928 08:57:33.229506 11116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance:
uuid: "dbd8629fa10e4963b308740f68d41ea5"
format_stamp: "Formatted at 2026-09-28 08:57:33 on dist-test-slave-pbjp"
server_key: "f81cfd08da2354e019e5402a4427be47"
server_key_iv: "ab9bbac788ca0204c48f1e9c7568f133"
server_key_version: "encryptionkey@0"
I20260928 08:57:33.230695 11116 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:57:33.231361 11131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:33.231527 11116 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:57:33.231606 11116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "dbd8629fa10e4963b308740f68d41ea5"
format_stamp: "Formatted at 2026-09-28 08:57:33 on dist-test-slave-pbjp"
server_key: "f81cfd08da2354e019e5402a4427be47"
server_key_iv: "ab9bbac788ca0204c48f1e9c7568f133"
server_key_version: "encryptionkey@0"
I20260928 08:57:33.231659 11116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:33.242713 11116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:33.244314 11116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:33.244447 11116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:33.244657 11116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:57:33.244935 11116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:57:33.244966 11116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:33.244987 11116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:57:33.245011 11116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:33.250377 11116 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:33667
I20260928 08:57:33.250461 11244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:33667 every 8 connection(s)
I20260928 08:57:33.250725 11116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
Sep 28 08:57:33 dist-test-slave-pbjp krb5kdc[10875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585853, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.194@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:57:33.259516 10978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:33.251445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:59953 (local address 127.23.167.254:37607)
0928 08:57:33.251499 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:33.251501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:33.251704 (+ 203us) server_negotiation.cc:407] Connection header received
0928 08:57:33.252210 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:33.252212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:33.252222 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:33.252284 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:33.253257 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.253780 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.254771 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.254973 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.255026 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:33.257044 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:33.257046 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:33.257046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:33.257055 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:33.258391 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:33.258763 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:33.258764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:33.258764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:33.258819 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:33.259013 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:33.259014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:33.259014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:33.259115 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:33.259161 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:33.259379 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 08:57:33.259448 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 08:57:33.259687 11247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:33.251425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:37607 (local address 127.23.167.194:59953)
0928 08:57:33.251580 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:33.251622 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:33.252098 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:33.252341 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:33.252344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:33.252913 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:57:33.253160 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:33.253168 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.253884 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.253884 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:33.254695 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:33.254698 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.255035 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.255035 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:33.255135 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:33.255585 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:57:33.255594 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:57:33.256950 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:57:33.258474 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:33.258481 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:33.258483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:33.258698 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:33.258869 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:33.258869 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:33.258870 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:33.258962 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:33.259157 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:57:33.259160 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:57:33.259272 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 08:57:33.259364 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 08:57:33.259482 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260928 08:57:33.259789 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 11116
I20260928 08:57:33.259898 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance
I20260928 08:57:33.260074 24223 external_mini_cluster.cc:1428] Setting key d236d722f0097eca33cf6a006e0d946d
I20260928 08:57:33.260118 11245 heartbeater.cc:347] Connected to a master server at 127.23.167.254:37607
I20260928 08:57:33.260214 11245 heartbeater.cc:464] Registering TS with master...
I20260928 08:57:33.260391 11245 heartbeater.cc:511] Master 127.23.167.254:37607 requested a full tablet report, sending...
I20260928 08:57:33.260805 10920 ts_manager.cc:194] Registered new tserver with Master: dbd8629fa10e4963b308740f68d41ea5 (127.23.167.194:33667)
I20260928 08:57:33.261282 10920 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.194@KRBTEST.COM'} at 127.23.167.194:59953
WARNING: no policy specified for kudu/127.23.167.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
Entry for principal kudu/127.23.167.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab.
I20260928 08:57:33.283174 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:0
--local_ip_for_outbound_sockets=127.23.167.195
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:37607
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:33.363994 11250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:33.364189 11250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:33.364212 11250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:33.365316 11250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:33.365355 11250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:33.365379 11250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:57:33.366757 11250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.195.keytab
--principal=kudu/127.23.167.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:37607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:33.366941 11250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:33.367125 11250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:33.371758 11255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:33.371801 11258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:33.371758 11256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:33.372016 11250 server_base.cc:1034] running on GCE node
I20260928 08:57:33.372221 11250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:33.372478 11250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:33.373621 11250 hybrid_clock.cc:648] HybridClock initialized: now 1790585853373600 us; error 34 us; skew 500 ppm
Sep 28 08:57:33 dist-test-slave-pbjp krb5kdc[10875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585853, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:33.375970 11250 init.cc:377] Logged in from keytab as kudu/127.23.167.195@KRBTEST.COM (short username kudu)
I20260928 08:57:33.376351 11250 webserver.cc:460] Webserver started at http://127.23.167.195:36583/ using document root <none> and password file <none>
I20260928 08:57:33.376587 11250 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:33.376636 11250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:33.376754 11250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:33.377774 11250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-2/data/instance:
uuid: "98401735448a4c92820fbb995c2954f3"
format_stamp: "Formatted at 2026-09-28 08:57:33 on dist-test-slave-pbjp"
server_key: "64b00c8cee2f01a5c427e034f88f2e25"
server_key_iv: "a7979660338ad0ff4bbcd0df9db44523"
server_key_version: "encryptionkey@0"
I20260928 08:57:33.378083 11250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance:
uuid: "98401735448a4c92820fbb995c2954f3"
format_stamp: "Formatted at 2026-09-28 08:57:33 on dist-test-slave-pbjp"
server_key: "64b00c8cee2f01a5c427e034f88f2e25"
server_key_iv: "a7979660338ad0ff4bbcd0df9db44523"
server_key_version: "encryptionkey@0"
I20260928 08:57:33.379259 11250 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 08:57:33.379915 11265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:33.380070 11250 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:57:33.380149 11250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "98401735448a4c92820fbb995c2954f3"
format_stamp: "Formatted at 2026-09-28 08:57:33 on dist-test-slave-pbjp"
server_key: "64b00c8cee2f01a5c427e034f88f2e25"
server_key_iv: "a7979660338ad0ff4bbcd0df9db44523"
server_key_version: "encryptionkey@0"
I20260928 08:57:33.380208 11250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:33.389204 11250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:33.391165 11250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:33.391292 11250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:33.391486 11250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:57:33.391768 11250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:57:33.391800 11250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:33.391827 11250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:57:33.391857 11250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:33.397078 11250 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:40383
I20260928 08:57:33.397136 11378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:40383 every 8 connection(s)
I20260928 08:57:33.397418 11250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
I20260928 08:57:33.397643 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 11250
I20260928 08:57:33.397742 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance
I20260928 08:57:33.397895 24223 external_mini_cluster.cc:1428] Setting key 4e9a26a6c4052b8fee0dca1ed2a5040f
Sep 28 08:57:33 dist-test-slave-pbjp krb5kdc[10875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585853, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.195@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:57:33.405938 10978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:33.398176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:42569 (local address 127.23.167.254:37607)
0928 08:57:33.398222 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:33.398225 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:33.398486 (+ 261us) server_negotiation.cc:407] Connection header received
0928 08:57:33.399048 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:33.399049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:33.399060 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:33.399105 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:33.400024 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.400521 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.401555 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.401825 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.401888 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:33.403769 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:33.403771 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:33.403771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:33.403778 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:33.404806 (+ 1028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:33.405175 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:33.405176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:33.405176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:33.405224 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:33.405432 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:33.405432 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:33.405433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:33.405552 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:33.405602 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:33.405813 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 08:57:33.405873 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 08:57:33.406114 11381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:33.398171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:37607 (local address 127.23.167.195:42569)
0928 08:57:33.398373 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:33.398420 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:33.398963 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:33.399172 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:33.399176 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:33.399735 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:57:33.399933 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:33.399938 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.400597 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.400598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:33.401478 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:33.401481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.401900 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.401901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:33.402029 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:33.402413 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:57:33.402421 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:57:33.403689 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:57:33.404869 (+ 1180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:33.404875 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:33.404876 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:33.405122 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:33.405281 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:33.405282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:33.405282 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:33.405386 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:33.405600 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:57:33.405605 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:57:33.405719 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 08:57:33.405795 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 08:57:33.405926 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":78,"threads_started":1}
I20260928 08:57:33.406527 11379 heartbeater.cc:347] Connected to a master server at 127.23.167.254:37607
I20260928 08:57:33.406637 11379 heartbeater.cc:464] Registering TS with master...
I20260928 08:57:33.406805 11379 heartbeater.cc:511] Master 127.23.167.254:37607 requested a full tablet report, sending...
I20260928 08:57:33.407210 10920 ts_manager.cc:194] Registered new tserver with Master: 98401735448a4c92820fbb995c2954f3 (127.23.167.195:40383)
I20260928 08:57:33.407586 10920 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.167.195@KRBTEST.COM'} at 127.23.167.195:42569
I20260928 08:57:33.409583 24223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:57:33.416270 10978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:33.410768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59954 (local address 127.23.167.254:37607)
0928 08:57:33.410829 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:33.410831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:33.410957 (+ 126us) server_negotiation.cc:407] Connection header received
0928 08:57:33.410977 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:33.410978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:33.410988 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:33.411028 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:33.411513 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.411992 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.412686 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.412873 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.412924 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:33.413977 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:33.413979 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:33.413979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:33.413985 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:33.415251 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:33.415546 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:33.415547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:33.415547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:33.415594 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:33.415762 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:33.415764 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:33.415764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:33.415927 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:33.416023 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:33.416106 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 08:57:33.416171 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 08:57:33.417788 10920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:57:33.419407 10920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:57:33.425915 11387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:33.421567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:40383 (local address 127.0.0.1:46232)
0928 08:57:33.421842 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:33.421865 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:33.421913 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:33.422578 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:33.422579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:33.422581 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:57:33.422639 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:33.422640 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.423798 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.423799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:33.424681 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:33.424683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.425579 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.425580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:33.425665 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:33.425682 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 08:57:33.425746 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 08:57:33.425804 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":53,"threads_started":1}
I20260928 08:57:33.426023 11389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:33.421689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37426 (local address 127.23.167.194:33667)
0928 08:57:33.421984 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:33.421990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:33.422001 (+ 11us) server_negotiation.cc:407] Connection header received
0928 08:57:33.422062 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:33.422064 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:33.422079 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:33.422156 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:57:33.422304 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.423427 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.424944 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.425673 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.425736 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:33.425790 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:33.425901 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 08:57:33.425939 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":94,"threads_started":1}
I20260928 08:57:33.426033 11388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:33.421554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:33667 (local address 127.0.0.1:37426)
0928 08:57:33.421719 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:33.421759 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:33.421828 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:33.422160 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:33.422163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:33.422165 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:57:33.422227 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:33.422229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.423558 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.423559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:33.424862 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:33.424865 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.425748 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.425749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:33.425827 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:33.425832 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 08:57:33.425884 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 08:57:33.425929 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":44,"thread_start_us":59,"threads_started":1}
I20260928 08:57:33.426213 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:33.421955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52534 (local address 127.23.167.193:43603)
0928 08:57:33.422108 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:33.422112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:33.422358 (+ 246us) server_negotiation.cc:407] Connection header received
0928 08:57:33.422404 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:33.422406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:33.422422 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:33.422495 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:57:33.422711 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.423408 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.425128 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.425662 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.425744 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:33.425800 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:33.426084 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 08:57:33.426134 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":82,"threads_started":1}
I20260928 08:57:33.426044 11390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:33.421736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:43603 (local address 127.0.0.1:52534)
0928 08:57:33.422240 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:33.422248 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:33.422295 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:33.422602 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:33.422603 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:33.422605 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:57:33.422645 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:33.422646 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.423504 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.423505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:33.424862 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:33.424865 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.425832 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.425833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:33.425900 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:33.425903 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:57:33.425949 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 08:57:33.425987 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":36,"thread_start_us":57,"threads_started":1}
I20260928 08:57:33.426815 11391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:33.421729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46232 (local address 127.23.167.195:40383)
0928 08:57:33.422416 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:33.422420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:33.422429 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:57:33.422463 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:33.422465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:33.422479 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:33.422702 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:57:33.422762 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.423718 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.424757 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.425495 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.425571 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:33.425630 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:33.426677 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0928 08:57:33.426727 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":93,"threads_started":1}
I20260928 08:57:33.426748 11179 tablet_service.cc:1467] Processing CreateTablet for tablet 28f05868b06445abb3140a6b7f05642f (DEFAULT_TABLE table=test-table [id=b3a84d05746a4b148e047c8fbc1ac07e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:57:33.427074 11179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28f05868b06445abb3140a6b7f05642f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:33.426998 11045 tablet_service.cc:1467] Processing CreateTablet for tablet 28f05868b06445abb3140a6b7f05642f (DEFAULT_TABLE table=test-table [id=b3a84d05746a4b148e047c8fbc1ac07e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:57:33.427307 11045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28f05868b06445abb3140a6b7f05642f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:33.429718 11313 tablet_service.cc:1467] Processing CreateTablet for tablet 28f05868b06445abb3140a6b7f05642f (DEFAULT_TABLE table=test-table [id=b3a84d05746a4b148e047c8fbc1ac07e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:57:33.430027 11313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28f05868b06445abb3140a6b7f05642f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:33.430027 11393 tablet_bootstrap.cc:492] T 28f05868b06445abb3140a6b7f05642f P f86ecc27b4ab485897f78169ccb8e6de: Bootstrap starting.
I20260928 08:57:33.430724 11393 tablet_bootstrap.cc:654] T 28f05868b06445abb3140a6b7f05642f P f86ecc27b4ab485897f78169ccb8e6de: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:33.431319 11393 log.cc:824] T 28f05868b06445abb3140a6b7f05642f P f86ecc27b4ab485897f78169ccb8e6de: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:33.432214 11393 tablet_bootstrap.cc:492] T 28f05868b06445abb3140a6b7f05642f P f86ecc27b4ab485897f78169ccb8e6de: No bootstrap required, opened a new log
I20260928 08:57:33.432273 11393 ts_tablet_manager.cc:1397] T 28f05868b06445abb3140a6b7f05642f P f86ecc27b4ab485897f78169ccb8e6de: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:57:33.433267 11395 tablet_bootstrap.cc:492] T 28f05868b06445abb3140a6b7f05642f P 98401735448a4c92820fbb995c2954f3: Bootstrap starting.
I20260928 08:57:33.433930 11395 tablet_bootstrap.cc:654] T 28f05868b06445abb3140a6b7f05642f P 98401735448a4c92820fbb995c2954f3: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:33.434114 11393 raft_consensus.cc:348] T 28f05868b06445abb3140a6b7f05642f P f86ecc27b4ab485897f78169ccb8e6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd8629fa10e4963b308740f68d41ea5" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33667 } } peers { permanent_uuid: "f86ecc27b4ab485897f78169ccb8e6de" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 43603 } } peers { permanent_uuid: "98401735448a4c92820fbb995c2954f3" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 40383 } }
I20260928 08:57:33.434312 11393 raft_consensus.cc:374] T 28f05868b06445abb3140a6b7f05642f P f86ecc27b4ab485897f78169ccb8e6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:33.434343 11393 raft_consensus.cc:729] T 28f05868b06445abb3140a6b7f05642f P f86ecc27b4ab485897f78169ccb8e6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f86ecc27b4ab485897f78169ccb8e6de, State: Initialized, Role: FOLLOWER
I20260928 08:57:33.434458 11393 consensus_queue.cc:260] T 28f05868b06445abb3140a6b7f05642f P f86ecc27b4ab485897f78169ccb8e6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd8629fa10e4963b308740f68d41ea5" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33667 } } peers { permanent_uuid: "f86ecc27b4ab485897f78169ccb8e6de" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 43603 } } peers { permanent_uuid: "98401735448a4c92820fbb995c2954f3" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 40383 } }
I20260928 08:57:33.434607 11395 log.cc:824] T 28f05868b06445abb3140a6b7f05642f P 98401735448a4c92820fbb995c2954f3: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:33.434729 11393 ts_tablet_manager.cc:1428] T 28f05868b06445abb3140a6b7f05642f P f86ecc27b4ab485897f78169ccb8e6de: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 08:57:33.434741 11111 heartbeater.cc:503] Master 127.23.167.254:37607 was elected leader, sending a full tablet report...
I20260928 08:57:33.434953 11396 tablet_bootstrap.cc:492] T 28f05868b06445abb3140a6b7f05642f P dbd8629fa10e4963b308740f68d41ea5: Bootstrap starting.
I20260928 08:57:33.435449 11395 tablet_bootstrap.cc:492] T 28f05868b06445abb3140a6b7f05642f P 98401735448a4c92820fbb995c2954f3: No bootstrap required, opened a new log
I20260928 08:57:33.435515 11395 ts_tablet_manager.cc:1397] T 28f05868b06445abb3140a6b7f05642f P 98401735448a4c92820fbb995c2954f3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:57:33.435619 11396 tablet_bootstrap.cc:654] T 28f05868b06445abb3140a6b7f05642f P dbd8629fa10e4963b308740f68d41ea5: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:33.436192 11396 log.cc:824] T 28f05868b06445abb3140a6b7f05642f P dbd8629fa10e4963b308740f68d41ea5: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:33.436825 11396 tablet_bootstrap.cc:492] T 28f05868b06445abb3140a6b7f05642f P dbd8629fa10e4963b308740f68d41ea5: No bootstrap required, opened a new log
I20260928 08:57:33.436885 11396 ts_tablet_manager.cc:1397] T 28f05868b06445abb3140a6b7f05642f P dbd8629fa10e4963b308740f68d41ea5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:57:33.437171 11395 raft_consensus.cc:348] T 28f05868b06445abb3140a6b7f05642f P 98401735448a4c92820fbb995c2954f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd8629fa10e4963b308740f68d41ea5" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33667 } } peers { permanent_uuid: "f86ecc27b4ab485897f78169ccb8e6de" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 43603 } } peers { permanent_uuid: "98401735448a4c92820fbb995c2954f3" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 40383 } }
I20260928 08:57:33.437311 11395 raft_consensus.cc:374] T 28f05868b06445abb3140a6b7f05642f P 98401735448a4c92820fbb995c2954f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:33.437332 11395 raft_consensus.cc:729] T 28f05868b06445abb3140a6b7f05642f P 98401735448a4c92820fbb995c2954f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98401735448a4c92820fbb995c2954f3, State: Initialized, Role: FOLLOWER
I20260928 08:57:33.437443 11395 consensus_queue.cc:260] T 28f05868b06445abb3140a6b7f05642f P 98401735448a4c92820fbb995c2954f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd8629fa10e4963b308740f68d41ea5" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33667 } } peers { permanent_uuid: "f86ecc27b4ab485897f78169ccb8e6de" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 43603 } } peers { permanent_uuid: "98401735448a4c92820fbb995c2954f3" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 40383 } }
I20260928 08:57:33.437676 11395 ts_tablet_manager.cc:1428] T 28f05868b06445abb3140a6b7f05642f P 98401735448a4c92820fbb995c2954f3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:57:33.437711 11379 heartbeater.cc:503] Master 127.23.167.254:37607 was elected leader, sending a full tablet report...
I20260928 08:57:33.438650 11396 raft_consensus.cc:348] T 28f05868b06445abb3140a6b7f05642f P dbd8629fa10e4963b308740f68d41ea5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd8629fa10e4963b308740f68d41ea5" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33667 } } peers { permanent_uuid: "f86ecc27b4ab485897f78169ccb8e6de" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 43603 } } peers { permanent_uuid: "98401735448a4c92820fbb995c2954f3" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 40383 } }
I20260928 08:57:33.438800 11396 raft_consensus.cc:374] T 28f05868b06445abb3140a6b7f05642f P dbd8629fa10e4963b308740f68d41ea5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:33.438832 11396 raft_consensus.cc:729] T 28f05868b06445abb3140a6b7f05642f P dbd8629fa10e4963b308740f68d41ea5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbd8629fa10e4963b308740f68d41ea5, State: Initialized, Role: FOLLOWER
I20260928 08:57:33.438942 11396 consensus_queue.cc:260] T 28f05868b06445abb3140a6b7f05642f P dbd8629fa10e4963b308740f68d41ea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd8629fa10e4963b308740f68d41ea5" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33667 } } peers { permanent_uuid: "f86ecc27b4ab485897f78169ccb8e6de" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 43603 } } peers { permanent_uuid: "98401735448a4c92820fbb995c2954f3" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 40383 } }
I20260928 08:57:33.439189 11396 ts_tablet_manager.cc:1428] T 28f05868b06445abb3140a6b7f05642f P dbd8629fa10e4963b308740f68d41ea5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 08:57:33.439250 11245 heartbeater.cc:503] Master 127.23.167.254:37607 was elected leader, sending a full tablet report...
W20260928 08:57:33.501237 11246 tablet.cc:2369] T 28f05868b06445abb3140a6b7f05642f P dbd8629fa10e4963b308740f68d41ea5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:57:33.502595 11397 raft_consensus.cc:482] T 28f05868b06445abb3140a6b7f05642f P f86ecc27b4ab485897f78169ccb8e6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:57:33.502703 11397 raft_consensus.cc:504] T 28f05868b06445abb3140a6b7f05642f P f86ecc27b4ab485897f78169ccb8e6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd8629fa10e4963b308740f68d41ea5" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33667 } } peers { permanent_uuid: "f86ecc27b4ab485897f78169ccb8e6de" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 43603 } } peers { permanent_uuid: "98401735448a4c92820fbb995c2954f3" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 40383 } }
I20260928 08:57:33.503067 11397 leader_election.cc:288] T 28f05868b06445abb3140a6b7f05642f P f86ecc27b4ab485897f78169ccb8e6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbd8629fa10e4963b308740f68d41ea5 (127.23.167.194:33667), 98401735448a4c92820fbb995c2954f3 (127.23.167.195:40383)
I20260928 08:57:33.505985 11391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:33.503339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:40231 (local address 127.23.167.195:40383)
0928 08:57:33.503382 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:33.503383 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:33.503534 (+ 151us) server_negotiation.cc:407] Connection header received
0928 08:57:33.503565 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:33.503566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:33.503575 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:33.503635 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:57:33.503790 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.504241 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.505230 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.505649 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.505717 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:33.505726 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:33.505896 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 08:57:33.505931 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 08:57:33.506019 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:33.503238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:33667 (local address 127.23.167.193:55671)
0928 08:57:33.503312 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:33.503325 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:33.503384 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:33.503539 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:33.503539 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:33.503541 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:57:33.503629 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:33.503630 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.504272 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.504273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:33.505163 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:33.505166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.505736 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.505738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:33.505818 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:33.505822 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:57:33.505874 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 08:57:33.505920 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":28}
I20260928 08:57:33.506024 11402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:33.503257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:40383 (local address 127.23.167.193:40231)
0928 08:57:33.503451 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:33.503461 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:33.503523 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:33.503637 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:33.503638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:33.503639 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:57:33.503729 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:33.503730 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.504317 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.504318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:33.505149 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:33.505152 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.505754 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.505755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:33.505818 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:33.505824 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 08:57:33.505880 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 08:57:33.505919 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":4,"thread_start_us":120,"threads_started":1}
I20260928 08:57:33.506196 11389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:33.503307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:55671 (local address 127.23.167.194:33667)
0928 08:57:33.503351 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:33.503353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:33.503363 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:57:33.503456 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:33.503457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:33.503467 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:33.503528 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:57:33.503694 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.504198 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.505252 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.505639 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.505736 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:33.505745 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:33.506083 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 08:57:33.506129 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 08:57:33.506433 11333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28f05868b06445abb3140a6b7f05642f" candidate_uuid: "f86ecc27b4ab485897f78169ccb8e6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98401735448a4c92820fbb995c2954f3" is_pre_election: true
I20260928 08:57:33.506479 11199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28f05868b06445abb3140a6b7f05642f" candidate_uuid: "f86ecc27b4ab485897f78169ccb8e6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd8629fa10e4963b308740f68d41ea5" is_pre_election: true
I20260928 08:57:33.506598 11199 raft_consensus.cc:2454] T 28f05868b06445abb3140a6b7f05642f P dbd8629fa10e4963b308740f68d41ea5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f86ecc27b4ab485897f78169ccb8e6de in term 0.
I20260928 08:57:33.506599 11333 raft_consensus.cc:2454] T 28f05868b06445abb3140a6b7f05642f P 98401735448a4c92820fbb995c2954f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f86ecc27b4ab485897f78169ccb8e6de in term 0.
I20260928 08:57:33.506840 11000 leader_election.cc:302] T 28f05868b06445abb3140a6b7f05642f P f86ecc27b4ab485897f78169ccb8e6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98401735448a4c92820fbb995c2954f3, f86ecc27b4ab485897f78169ccb8e6de; no voters:
I20260928 08:57:33.507011 11397 raft_consensus.cc:2781] T 28f05868b06445abb3140a6b7f05642f P f86ecc27b4ab485897f78169ccb8e6de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:57:33.507066 11397 raft_consensus.cc:482] T 28f05868b06445abb3140a6b7f05642f P f86ecc27b4ab485897f78169ccb8e6de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:57:33.507081 11397 raft_consensus.cc:3037] T 28f05868b06445abb3140a6b7f05642f P f86ecc27b4ab485897f78169ccb8e6de [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:33.507774 11397 raft_consensus.cc:504] T 28f05868b06445abb3140a6b7f05642f P f86ecc27b4ab485897f78169ccb8e6de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd8629fa10e4963b308740f68d41ea5" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33667 } } peers { permanent_uuid: "f86ecc27b4ab485897f78169ccb8e6de" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 43603 } } peers { permanent_uuid: "98401735448a4c92820fbb995c2954f3" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 40383 } }
I20260928 08:57:33.507900 11397 leader_election.cc:288] T 28f05868b06445abb3140a6b7f05642f P f86ecc27b4ab485897f78169ccb8e6de [CANDIDATE]: Term 1 election: Requested vote from peers dbd8629fa10e4963b308740f68d41ea5 (127.23.167.194:33667), 98401735448a4c92820fbb995c2954f3 (127.23.167.195:40383)
I20260928 08:57:33.508064 11333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28f05868b06445abb3140a6b7f05642f" candidate_uuid: "f86ecc27b4ab485897f78169ccb8e6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98401735448a4c92820fbb995c2954f3"
I20260928 08:57:33.508071 11199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28f05868b06445abb3140a6b7f05642f" candidate_uuid: "f86ecc27b4ab485897f78169ccb8e6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd8629fa10e4963b308740f68d41ea5"
I20260928 08:57:33.508133 11333 raft_consensus.cc:3037] T 28f05868b06445abb3140a6b7f05642f P 98401735448a4c92820fbb995c2954f3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:33.508148 11199 raft_consensus.cc:3037] T 28f05868b06445abb3140a6b7f05642f P dbd8629fa10e4963b308740f68d41ea5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:33.508807 11333 raft_consensus.cc:2454] T 28f05868b06445abb3140a6b7f05642f P 98401735448a4c92820fbb995c2954f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f86ecc27b4ab485897f78169ccb8e6de in term 1.
I20260928 08:57:33.508905 11199 raft_consensus.cc:2454] T 28f05868b06445abb3140a6b7f05642f P dbd8629fa10e4963b308740f68d41ea5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f86ecc27b4ab485897f78169ccb8e6de in term 1.
I20260928 08:57:33.508973 11000 leader_election.cc:302] T 28f05868b06445abb3140a6b7f05642f P f86ecc27b4ab485897f78169ccb8e6de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98401735448a4c92820fbb995c2954f3, f86ecc27b4ab485897f78169ccb8e6de; no voters:
I20260928 08:57:33.509178 11397 raft_consensus.cc:2781] T 28f05868b06445abb3140a6b7f05642f P f86ecc27b4ab485897f78169ccb8e6de [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:57:33.509356 11397 raft_consensus.cc:686] T 28f05868b06445abb3140a6b7f05642f P f86ecc27b4ab485897f78169ccb8e6de [term 1 LEADER]: Becoming Leader. State: Replica: f86ecc27b4ab485897f78169ccb8e6de, State: Running, Role: LEADER
I20260928 08:57:33.509460 11397 consensus_queue.cc:237] T 28f05868b06445abb3140a6b7f05642f P f86ecc27b4ab485897f78169ccb8e6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd8629fa10e4963b308740f68d41ea5" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33667 } } peers { permanent_uuid: "f86ecc27b4ab485897f78169ccb8e6de" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 43603 } } peers { permanent_uuid: "98401735448a4c92820fbb995c2954f3" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 40383 } }
I20260928 08:57:33.510370 10919 catalog_manager.cc:5495] T 28f05868b06445abb3140a6b7f05642f P f86ecc27b4ab485897f78169ccb8e6de reported cstate change: term changed from 0 to 1, leader changed from <none> to f86ecc27b4ab485897f78169ccb8e6de (127.23.167.193). New cstate: current_term: 1 leader_uuid: "f86ecc27b4ab485897f78169ccb8e6de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd8629fa10e4963b308740f68d41ea5" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f86ecc27b4ab485897f78169ccb8e6de" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98401735448a4c92820fbb995c2954f3" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 40383 } health_report { overall_health: UNKNOWN } } }
I20260928 08:57:33.520409 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:33.518038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52540 (local address 127.23.167.193:43603)
0928 08:57:33.518088 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:33.518089 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:33.518097 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:57:33.518171 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:33.518172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:33.518189 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:33.518251 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:57:33.518416 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.518873 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.519616 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.519894 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.519951 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:33.520105 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:57:33.520216 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:57:33.520288 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:33.520307 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 08:57:33.520327 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 08:57:33.522185 11199 raft_consensus.cc:1261] T 28f05868b06445abb3140a6b7f05642f P dbd8629fa10e4963b308740f68d41ea5 [term 1 FOLLOWER]: Refusing update from remote peer f86ecc27b4ab485897f78169ccb8e6de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:57:33.522185 11333 raft_consensus.cc:1261] T 28f05868b06445abb3140a6b7f05642f P 98401735448a4c92820fbb995c2954f3 [term 1 FOLLOWER]: Refusing update from remote peer f86ecc27b4ab485897f78169ccb8e6de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:57:33.522549 11397 consensus_queue.cc:1035] T 28f05868b06445abb3140a6b7f05642f P f86ecc27b4ab485897f78169ccb8e6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "98401735448a4c92820fbb995c2954f3" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 40383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:57:33.522707 11403 consensus_queue.cc:1035] T 28f05868b06445abb3140a6b7f05642f P f86ecc27b4ab485897f78169ccb8e6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbd8629fa10e4963b308740f68d41ea5" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 33667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:57:33.525662 11408 mvcc.cc:205] Tried to move back new op lower bound from 7334239656024252416 to 7334239655975583744. Current Snapshot: MvccSnapshot[applied={T|T < 7334239656024252416}]
I20260928 08:57:33.525941 11407 mvcc.cc:205] Tried to move back new op lower bound from 7334239656024252416 to 7334239655975583744. Current Snapshot: MvccSnapshot[applied={T|T < 7334239656024252416 or (T in {7334239656024252416})}]
I20260928 08:57:33.527451 10919 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:57:33.527534 10919 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:57:33.528254 10919 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 41461c9167a24a3c8b725ce8adbce38c: Sending DeleteTablet for 3 replicas of tablet 28f05868b06445abb3140a6b7f05642f
I20260928 08:57:33.528524 11179 tablet_service.cc:1514] Processing DeleteTablet for tablet 28f05868b06445abb3140a6b7f05642f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:57:33 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:37426
I20260928 08:57:33.528653 11313 tablet_service.cc:1514] Processing DeleteTablet for tablet 28f05868b06445abb3140a6b7f05642f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:57:33 UTC) from {username='kudu', principal='kudu/127.23.167.254@KRBTEST.COM'} at 127.0.0.1:46232
I20260928 08:57:33.528862 11417 tablet_replica.cc:330] T 28f05868b06445abb3140a6b7f05642f P 98401735448a4c92820fbb995c2954f3: stopping tablet replica
I20260928 08:57:33.528900 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 10982
I20260928 08:57:33.528985 11417 raft_consensus.cc:2229] T 28f05868b06445abb3140a6b7f05642f P 98401735448a4c92820fbb995c2954f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:57:33.529122 11417 raft_consensus.cc:2258] T 28f05868b06445abb3140a6b7f05642f P 98401735448a4c92820fbb995c2954f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:57:33.529700 11417 ts_tablet_manager.cc:1905] T 28f05868b06445abb3140a6b7f05642f P 98401735448a4c92820fbb995c2954f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:57:33.531206 11417 ts_tablet_manager.cc:1918] T 28f05868b06445abb3140a6b7f05642f P 98401735448a4c92820fbb995c2954f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:57:33.531256 11417 log.cc:1196] T 28f05868b06445abb3140a6b7f05642f P 98401735448a4c92820fbb995c2954f3: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-2/wal/wals/28f05868b06445abb3140a6b7f05642f
I20260928 08:57:33.531464 11417 ts_tablet_manager.cc:1939] T 28f05868b06445abb3140a6b7f05642f P 98401735448a4c92820fbb995c2954f3: Deleting consensus metadata
I20260928 08:57:33.531798 10906 catalog_manager.cc:4841] TS 98401735448a4c92820fbb995c2954f3 (127.23.167.195:40383): tablet 28f05868b06445abb3140a6b7f05642f (table test-table [id=b3a84d05746a4b148e047c8fbc1ac07e]) successfully deleted
I20260928 08:57:33.533177 11416 tablet_replica.cc:330] T 28f05868b06445abb3140a6b7f05642f P dbd8629fa10e4963b308740f68d41ea5: stopping tablet replica
I20260928 08:57:33.533284 11416 raft_consensus.cc:2229] T 28f05868b06445abb3140a6b7f05642f P dbd8629fa10e4963b308740f68d41ea5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:57:33.533393 11416 raft_consensus.cc:2258] T 28f05868b06445abb3140a6b7f05642f P dbd8629fa10e4963b308740f68d41ea5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:57:33.533748 11416 ts_tablet_manager.cc:1905] T 28f05868b06445abb3140a6b7f05642f P dbd8629fa10e4963b308740f68d41ea5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 08:57:33.535038 10905 connection.cc:535] client connection to 127.23.167.193:43603 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 08:57:33.535063 11416 ts_tablet_manager.cc:1918] T 28f05868b06445abb3140a6b7f05642f P dbd8629fa10e4963b308740f68d41ea5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:57:33.535120 11416 log.cc:1196] T 28f05868b06445abb3140a6b7f05642f P dbd8629fa10e4963b308740f68d41ea5: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585589999956-24223-0/minicluster-data/ts-1/wal/wals/28f05868b06445abb3140a6b7f05642f
W20260928 08:57:33.535137 10905 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 08:57:33.535223 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 11116
I20260928 08:57:33.535991 11390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:33.535866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:43603 (local address 127.0.0.1:52546)
0928 08:57:33.535917 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:33.535955 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.193:43603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 08:57:33.536182 10905 catalog_manager.cc:4584] TS f86ecc27b4ab485897f78169ccb8e6de (127.23.167.193:43603): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 28f05868b06445abb3140a6b7f05642f: Network error: Client connection negotiation failed: client connection to 127.23.167.193:43603: connect: Connection refused (error 111)
I20260928 08:57:33.539693 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 11250
I20260928 08:57:33.539728 11390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:33.539621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:33667 (local address 127.0.0.1:37428)
0928 08:57:33.539670 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:33.539700 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.194:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 08:57:33.543818 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 10888
2026-09-28T08:57:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1783 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T08:57:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:57:33Z Disabled control of system clock
I20260928 08:57:33.569713 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:46343
--webserver_interface=127.23.167.254
--webserver_port=0
--builtin_ntp_servers=127.23.167.212:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:46343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 08:57:33.649722 11422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:33.649950 11422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:33.649971 11422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:33.651115 11422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:57:33.651153 11422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:33.651165 11422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:33.651175 11422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:57:33.651186 11422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:57:33.652549 11422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:46343
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:46343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:33.652777 11422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:33.653053 11422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:33.657799 11428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:33.657994 11422 server_base.cc:1034] running on GCE node
W20260928 08:57:33.657804 11427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:33.657799 11430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:33.658353 11422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:33.658679 11422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:33.659821 11422 hybrid_clock.cc:648] HybridClock initialized: now 1790585853659805 us; error 41 us; skew 500 ppm
I20260928 08:57:33.660951 11422 webserver.cc:460] Webserver started at http://127.23.167.254:36609/ using document root <none> and password file <none>
I20260928 08:57:33.661203 11422 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:33.661254 11422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:33.661378 11422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:33.662492 11422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "f3db08cbf9644e73917eb33b6d034f27"
format_stamp: "Formatted at 2026-09-28 08:57:33 on dist-test-slave-pbjp"
server_key: "291b56954c229c7637abd930a30c0028"
server_key_iv: "f1905fd2dc278c516597558725c1a03b"
server_key_version: "encryptionkey@0"
I20260928 08:57:33.662825 11422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "f3db08cbf9644e73917eb33b6d034f27"
format_stamp: "Formatted at 2026-09-28 08:57:33 on dist-test-slave-pbjp"
server_key: "291b56954c229c7637abd930a30c0028"
server_key_iv: "f1905fd2dc278c516597558725c1a03b"
server_key_version: "encryptionkey@0"
I20260928 08:57:33.664067 11422 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:57:33.664683 11436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:33.664846 11422 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:57:33.664923 11422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "f3db08cbf9644e73917eb33b6d034f27"
format_stamp: "Formatted at 2026-09-28 08:57:33 on dist-test-slave-pbjp"
server_key: "291b56954c229c7637abd930a30c0028"
server_key_iv: "f1905fd2dc278c516597558725c1a03b"
server_key_version: "encryptionkey@0"
I20260928 08:57:33.664983 11422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:33.686756 11422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:33.687077 11422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:33.687213 11422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:33.690948 11422 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:46343
I20260928 08:57:33.690989 11488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:46343 every 8 connection(s)
I20260928 08:57:33.691311 11422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:57:33.692284 11489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:33.693993 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 11422
I20260928 08:57:33.694099 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:57:33.694293 24223 external_mini_cluster.cc:1428] Setting key 03317cbf6608b65c1d81f31a89262a02
I20260928 08:57:33.695066 11489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3db08cbf9644e73917eb33b6d034f27: Bootstrap starting.
I20260928 08:57:33.695631 11489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3db08cbf9644e73917eb33b6d034f27: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:33.695940 11489 log.cc:824] T 00000000000000000000000000000000 P f3db08cbf9644e73917eb33b6d034f27: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:33.696611 11489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3db08cbf9644e73917eb33b6d034f27: No bootstrap required, opened a new log
I20260928 08:57:33.698796 11489 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3db08cbf9644e73917eb33b6d034f27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3db08cbf9644e73917eb33b6d034f27" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46343 } }
I20260928 08:57:33.698974 11489 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3db08cbf9644e73917eb33b6d034f27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:33.699013 11489 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3db08cbf9644e73917eb33b6d034f27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3db08cbf9644e73917eb33b6d034f27, State: Initialized, Role: FOLLOWER
I20260928 08:57:33.699177 11489 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3db08cbf9644e73917eb33b6d034f27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3db08cbf9644e73917eb33b6d034f27" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46343 } }
I20260928 08:57:33.699267 11489 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3db08cbf9644e73917eb33b6d034f27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:57:33.699301 11489 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3db08cbf9644e73917eb33b6d034f27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:57:33.699366 11489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3db08cbf9644e73917eb33b6d034f27 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:33.699975 11489 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3db08cbf9644e73917eb33b6d034f27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3db08cbf9644e73917eb33b6d034f27" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46343 } }
I20260928 08:57:33.699976 11492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:33.694882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40212 (local address 127.23.167.254:46343)
0928 08:57:33.695071 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:33.695079 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:33.695104 (+ 25us) server_negotiation.cc:407] Connection header received
0928 08:57:33.695539 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:33.695546 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:33.695610 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:33.695750 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:33.695977 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.696995 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.698132 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.698525 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.698613 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:33.699312 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:33.699327 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:33.699327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:33.699522 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:33.699557 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:33.699568 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:33.699614 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:33.699714 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 08:57:33.699777 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":63,"threads_started":1}
I20260928 08:57:33.700086 11489 leader_election.cc:302] T 00000000000000000000000000000000 P f3db08cbf9644e73917eb33b6d034f27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3db08cbf9644e73917eb33b6d034f27; no voters:
I20260928 08:57:33.700237 11489 leader_election.cc:288] T 00000000000000000000000000000000 P f3db08cbf9644e73917eb33b6d034f27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:57:33.700284 11494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3db08cbf9644e73917eb33b6d034f27 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:57:33.700419 11494 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3db08cbf9644e73917eb33b6d034f27 [term 1 LEADER]: Becoming Leader. State: Replica: f3db08cbf9644e73917eb33b6d034f27, State: Running, Role: LEADER
I20260928 08:57:33.700482 11489 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3db08cbf9644e73917eb33b6d034f27 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:57:33.700534 11494 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3db08cbf9644e73917eb33b6d034f27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3db08cbf9644e73917eb33b6d034f27" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46343 } }
I20260928 08:57:33.700913 11494 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3db08cbf9644e73917eb33b6d034f27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3db08cbf9644e73917eb33b6d034f27. Latest consensus state: current_term: 1 leader_uuid: "f3db08cbf9644e73917eb33b6d034f27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3db08cbf9644e73917eb33b6d034f27" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46343 } } }
I20260928 08:57:33.700937 11495 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3db08cbf9644e73917eb33b6d034f27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3db08cbf9644e73917eb33b6d034f27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3db08cbf9644e73917eb33b6d034f27" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46343 } } }
I20260928 08:57:33.700996 11494 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3db08cbf9644e73917eb33b6d034f27 [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:33.701004 11495 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3db08cbf9644e73917eb33b6d034f27 [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:33.701201 11502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:57:33.701632 11502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:57:33.703040 11502 catalog_manager.cc:1347] Generated new cluster ID: ef51c5ab10d24e2b8db471f4154564a9
I20260928 08:57:33.703094 11502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:57:33.709825 11502 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:57:33.710492 11502 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:57:33.715939 11502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3db08cbf9644e73917eb33b6d034f27: Generated new TSK 0
I20260928 08:57:33.716118 11502 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:57:33.720419 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.193:0
--local_ip_for_outbound_sockets=127.23.167.193
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:46343
--builtin_ntp_servers=127.23.167.212:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 08:57:33.803287 11513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:33.803496 11513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:33.803525 11513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:33.804661 11513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:33.804698 11513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:33.804731 11513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.193
I20260928 08:57:33.806119 11513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.167.193
--webserver_port=0
--tserver_master_addrs=127.23.167.254:46343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.193
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:33.806351 11513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:33.806630 11513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:33.811460 11518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:33.811594 11513 server_base.cc:1034] running on GCE node
W20260928 08:57:33.811530 11521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:33.811470 11519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:33.811893 11513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:33.812132 11513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:33.813271 11513 hybrid_clock.cc:648] HybridClock initialized: now 1790585853813257 us; error 28 us; skew 500 ppm
I20260928 08:57:33.814457 11513 webserver.cc:460] Webserver started at http://127.23.167.193:45947/ using document root <none> and password file <none>
I20260928 08:57:33.814683 11513 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:33.814733 11513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:33.814880 11513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:33.815915 11513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-0/data/instance:
uuid: "b84c9ebe6d1c4cd1aa5e899de71986dc"
format_stamp: "Formatted at 2026-09-28 08:57:33 on dist-test-slave-pbjp"
server_key: "0cc25ca6b864e95f6114ebbdb2dfd10a"
server_key_iv: "4c6e1808ddea8eb3af3cdba9c1b0ffb1"
server_key_version: "encryptionkey@0"
I20260928 08:57:33.816232 11513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance:
uuid: "b84c9ebe6d1c4cd1aa5e899de71986dc"
format_stamp: "Formatted at 2026-09-28 08:57:33 on dist-test-slave-pbjp"
server_key: "0cc25ca6b864e95f6114ebbdb2dfd10a"
server_key_iv: "4c6e1808ddea8eb3af3cdba9c1b0ffb1"
server_key_version: "encryptionkey@0"
I20260928 08:57:33.817557 11513 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 08:57:33.818311 11527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:33.818466 11513 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:57:33.818547 11513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-0/wal
uuid: "b84c9ebe6d1c4cd1aa5e899de71986dc"
format_stamp: "Formatted at 2026-09-28 08:57:33 on dist-test-slave-pbjp"
server_key: "0cc25ca6b864e95f6114ebbdb2dfd10a"
server_key_iv: "4c6e1808ddea8eb3af3cdba9c1b0ffb1"
server_key_version: "encryptionkey@0"
I20260928 08:57:33.818609 11513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:33.841885 11513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:33.842229 11513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:33.842366 11513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:33.842597 11513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:57:33.842926 11513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:57:33.842957 11513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:33.842988 11513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:57:33.843005 11513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:33.848272 11513 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.193:36805
I20260928 08:57:33.848325 11640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.193:36805 every 8 connection(s)
I20260928 08:57:33.848642 11513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-0/data/info.pb
I20260928 08:57:33.853720 11492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:33.849359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:55425 (local address 127.23.167.254:46343)
0928 08:57:33.849413 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:33.849415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:33.849634 (+ 219us) server_negotiation.cc:407] Connection header received
0928 08:57:33.850122 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:33.850123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:33.850137 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:33.850207 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:33.850888 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.851400 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.852435 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.852641 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.852702 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:33.853497 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:33.853498 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:33.853499 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:33.853503 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:33.853517 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:33.853521 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:33.853560 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:33.853634 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 08:57:33.853656 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 08:57:33.853919 11643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:33.849322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:46343 (local address 127.23.167.193:55425)
0928 08:57:33.849516 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:33.849558 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:33.850041 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:33.850269 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:33.850273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:33.850281 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:57:33.850781 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:33.850785 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.851523 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.851524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:33.852328 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:33.852330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.852721 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.852722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:33.852846 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:33.853329 (+ 483us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:57:33.853329 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:57:33.853393 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:57:33.853419 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:57:33.853575 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:57:33.853578 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:57:33.853581 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:57:33.853630 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 08:57:33.853719 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
I20260928 08:57:33.854382 11641 heartbeater.cc:347] Connected to a master server at 127.23.167.254:46343
I20260928 08:57:33.854477 11641 heartbeater.cc:464] Registering TS with master...
I20260928 08:57:33.854676 11641 heartbeater.cc:511] Master 127.23.167.254:46343 requested a full tablet report, sending...
I20260928 08:57:33.855201 11453 ts_manager.cc:194] Registered new tserver with Master: b84c9ebe6d1c4cd1aa5e899de71986dc (127.23.167.193:36805)
I20260928 08:57:33.855808 11453 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.167.193:55425
I20260928 08:57:33.858222 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 11513
I20260928 08:57:33.858314 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-0/wal/instance
I20260928 08:57:33.858482 24223 external_mini_cluster.cc:1428] Setting key 26e8768c924ec3754b3ec19798f5fb20
I20260928 08:57:33.859450 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.194:0
--local_ip_for_outbound_sockets=127.23.167.194
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:46343
--builtin_ntp_servers=127.23.167.212:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 08:57:33.938292 11644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:33.938512 11644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:33.938534 11644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:33.939656 11644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:33.939694 11644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:33.939718 11644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.194
I20260928 08:57:33.941021 11644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.167.194
--webserver_port=0
--tserver_master_addrs=127.23.167.254:46343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.194
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:33.941244 11644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:33.941519 11644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 08:57:33.946568 11644 server_base.cc:1034] running on GCE node
W20260928 08:57:33.946537 11652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:33.946530 11650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:33.946570 11649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:33.947018 11644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:33.947273 11644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:33.948431 11644 hybrid_clock.cc:648] HybridClock initialized: now 1790585853948418 us; error 30 us; skew 500 ppm
I20260928 08:57:33.949744 11644 webserver.cc:460] Webserver started at http://127.23.167.194:40227/ using document root <none> and password file <none>
I20260928 08:57:33.949973 11644 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:33.950032 11644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:33.950155 11644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:33.951242 11644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-1/data/instance:
uuid: "b89d2c1da8624b0cbb3aefc5c0e5882f"
format_stamp: "Formatted at 2026-09-28 08:57:33 on dist-test-slave-pbjp"
server_key: "f12822134fac88cc4d6f3697057ff102"
server_key_iv: "950ec28532c2c9e8ef2c15a59ae0ee61"
server_key_version: "encryptionkey@0"
I20260928 08:57:33.951570 11644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance:
uuid: "b89d2c1da8624b0cbb3aefc5c0e5882f"
format_stamp: "Formatted at 2026-09-28 08:57:33 on dist-test-slave-pbjp"
server_key: "f12822134fac88cc4d6f3697057ff102"
server_key_iv: "950ec28532c2c9e8ef2c15a59ae0ee61"
server_key_version: "encryptionkey@0"
I20260928 08:57:33.952824 11644 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:57:33.953589 11658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:33.953738 11644 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:33.953819 11644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-1/wal
uuid: "b89d2c1da8624b0cbb3aefc5c0e5882f"
format_stamp: "Formatted at 2026-09-28 08:57:33 on dist-test-slave-pbjp"
server_key: "f12822134fac88cc4d6f3697057ff102"
server_key_iv: "950ec28532c2c9e8ef2c15a59ae0ee61"
server_key_version: "encryptionkey@0"
I20260928 08:57:33.953889 11644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:33.968004 11644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:33.968317 11644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:33.968456 11644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:33.968688 11644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:57:33.969005 11644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:57:33.969038 11644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:33.969094 11644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:57:33.969118 11644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:33.974521 11644 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.194:41435
I20260928 08:57:33.974586 11771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.194:41435 every 8 connection(s)
I20260928 08:57:33.974895 11644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-1/data/info.pb
I20260928 08:57:33.979795 11492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:33.975691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.194:47545 (local address 127.23.167.254:46343)
0928 08:57:33.975755 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:33.975757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:33.976025 (+ 268us) server_negotiation.cc:407] Connection header received
0928 08:57:33.976462 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:33.976463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:33.976479 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:33.976532 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:33.977229 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.977718 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.978641 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.978817 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.978858 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:33.979589 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:33.979590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:33.979590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:33.979594 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:33.979609 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:33.979613 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:33.979652 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:33.979715 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 08:57:33.979736 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 08:57:33.979986 11774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:33.975658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:46343 (local address 127.23.167.194:47545)
0928 08:57:33.975855 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:33.975893 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:33.976378 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:33.976596 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:33.976600 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:33.976607 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:57:33.977148 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:33.977154 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.977784 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.977785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:33.978580 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:33.978582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:33.978871 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:33.978872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:33.978977 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:33.979462 (+ 485us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:57:33.979463 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:57:33.979491 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:57:33.979515 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:57:33.979659 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:57:33.979662 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:57:33.979664 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 08:57:33.979711 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 08:57:33.979789 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
I20260928 08:57:33.980399 11772 heartbeater.cc:347] Connected to a master server at 127.23.167.254:46343
I20260928 08:57:33.980501 11772 heartbeater.cc:464] Registering TS with master...
I20260928 08:57:33.980646 11772 heartbeater.cc:511] Master 127.23.167.254:46343 requested a full tablet report, sending...
I20260928 08:57:33.980975 11453 ts_manager.cc:194] Registered new tserver with Master: b89d2c1da8624b0cbb3aefc5c0e5882f (127.23.167.194:41435)
I20260928 08:57:33.981333 11453 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.167.194:47545
I20260928 08:57:33.984159 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 11644
I20260928 08:57:33.984246 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-1/wal/instance
I20260928 08:57:33.984423 24223 external_mini_cluster.cc:1428] Setting key db0208396586a2e667451cbd2f55db28
I20260928 08:57:33.985322 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.167.195:0
--local_ip_for_outbound_sockets=127.23.167.195
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:46343
--builtin_ntp_servers=127.23.167.212:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 08:57:34.064314 11775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:34.064539 11775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:34.064569 11775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:34.065760 11775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:34.065800 11775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:34.065833 11775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.167.195
I20260928 08:57:34.067226 11775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.167.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.167.195
--webserver_port=0
--tserver_master_addrs=127.23.167.254:46343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.167.195
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:34.067469 11775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:34.067762 11775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:34.072554 11783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:34.072566 11780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:34.072537 11781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:34.072733 11775 server_base.cc:1034] running on GCE node
I20260928 08:57:34.072993 11775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:34.073240 11775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:34.074374 11775 hybrid_clock.cc:648] HybridClock initialized: now 1790585854074388 us; error 51 us; skew 500 ppm
I20260928 08:57:34.075604 11775 webserver.cc:460] Webserver started at http://127.23.167.195:43943/ using document root <none> and password file <none>
I20260928 08:57:34.075829 11775 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:34.075877 11775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:34.075999 11775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:34.077019 11775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-2/data/instance:
uuid: "fa92de1c9dfb4b3797fe7aa315d8673e"
format_stamp: "Formatted at 2026-09-28 08:57:34 on dist-test-slave-pbjp"
server_key: "87f4b7ea769c59af91471ae9ca631a4c"
server_key_iv: "50459ad8ca7af15b0549fe7a56f47354"
server_key_version: "encryptionkey@0"
I20260928 08:57:34.077378 11775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance:
uuid: "fa92de1c9dfb4b3797fe7aa315d8673e"
format_stamp: "Formatted at 2026-09-28 08:57:34 on dist-test-slave-pbjp"
server_key: "87f4b7ea769c59af91471ae9ca631a4c"
server_key_iv: "50459ad8ca7af15b0549fe7a56f47354"
server_key_version: "encryptionkey@0"
I20260928 08:57:34.078590 11775 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:57:34.079269 11789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:34.079455 11775 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:57:34.079553 11775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-2/wal
uuid: "fa92de1c9dfb4b3797fe7aa315d8673e"
format_stamp: "Formatted at 2026-09-28 08:57:34 on dist-test-slave-pbjp"
server_key: "87f4b7ea769c59af91471ae9ca631a4c"
server_key_iv: "50459ad8ca7af15b0549fe7a56f47354"
server_key_version: "encryptionkey@0"
I20260928 08:57:34.079620 11775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:34.090123 11775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:34.090370 11775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:34.090495 11775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:34.090675 11775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:57:34.090965 11775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:57:34.090997 11775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:34.091025 11775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:57:34.091043 11775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:34.096371 11775 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.195:38407
I20260928 08:57:34.096428 11902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.195:38407 every 8 connection(s)
I20260928 08:57:34.096722 11775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-2/data/info.pb
I20260928 08:57:34.099622 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 11775
I20260928 08:57:34.099735 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-2/wal/instance
I20260928 08:57:34.099947 24223 external_mini_cluster.cc:1428] Setting key adde9dc05cb67385bb6d30c3e0493066
I20260928 08:57:34.102075 11492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:34.097378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.195:38233 (local address 127.23.167.254:46343)
0928 08:57:34.097426 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:34.097428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:34.097670 (+ 242us) server_negotiation.cc:407] Connection header received
0928 08:57:34.098164 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:34.098166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:34.098179 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:34.098230 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:34.098879 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:34.099409 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:34.100656 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:34.100931 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:34.100998 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:34.101809 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:34.101810 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:34.101811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:34.101815 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:34.101831 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:34.101835 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:34.101889 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:34.101963 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 08:57:34.101998 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 08:57:34.102277 11905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:34.097354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:46343 (local address 127.23.167.195:38233)
0928 08:57:34.097541 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:34.097585 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:34.098068 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:34.098299 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:34.098302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:34.098312 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:57:34.098790 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:34.098795 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:34.099501 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:34.099502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:34.100587 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:34.100590 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:34.100998 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:34.100999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:34.101168 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:34.101675 (+ 507us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:57:34.101676 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:57:34.101704 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:57:34.101727 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:57:34.101895 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:57:34.101900 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:57:34.101903 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:57:34.101966 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 08:57:34.102067 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":91,"threads_started":1}
I20260928 08:57:34.102692 11903 heartbeater.cc:347] Connected to a master server at 127.23.167.254:46343
I20260928 08:57:34.102783 11903 heartbeater.cc:464] Registering TS with master...
I20260928 08:57:34.102953 11903 heartbeater.cc:511] Master 127.23.167.254:46343 requested a full tablet report, sending...
I20260928 08:57:34.103273 11453 ts_manager.cc:194] Registered new tserver with Master: fa92de1c9dfb4b3797fe7aa315d8673e (127.23.167.195:38407)
I20260928 08:57:34.103649 11453 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.167.195:38233
I20260928 08:57:34.111616 24223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:57:34.115593 11492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:34.112941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40226 (local address 127.23.167.254:46343)
0928 08:57:34.113014 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:34.113016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:34.113081 (+ 65us) server_negotiation.cc:407] Connection header received
0928 08:57:34.113167 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:34.113167 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:34.113176 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:34.113213 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:34.113342 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:34.113816 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:34.114523 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:34.114695 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:34.114738 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:34.115352 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:34.115353 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:34.115353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:34.115357 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:34.115370 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:34.115374 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:34.115411 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:34.115502 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 08:57:34.115523 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 08:57:34.117015 11453 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40226:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:57:34.118661 11453 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:57:34.124814 11914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:34.120699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:41435 (local address 127.0.0.1:60350)
0928 08:57:34.121178 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:34.121222 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:34.121307 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:34.121511 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:34.121515 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:34.121517 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:57:34.121585 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:34.121587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:34.122604 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:34.122605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:34.123695 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:34.123697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:34.124533 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:34.124534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:34.124610 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:34.124624 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 08:57:34.124682 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 08:57:34.124736 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":81,"threads_started":1}
I20260928 08:57:34.124825 11915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:34.120792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60350 (local address 127.23.167.194:41435)
0928 08:57:34.121343 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:34.121348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:34.121360 (+ 12us) server_negotiation.cc:407] Connection header received
0928 08:57:34.121400 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:34.121402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:34.121430 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:34.121504 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:57:34.121644 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:34.122537 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:34.123782 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:34.124459 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:34.124519 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:34.124612 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:34.124705 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 08:57:34.124740 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":128,"threads_started":1}
I20260928 08:57:34.125142 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:34.120708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:38407 (local address 127.0.0.1:53832)
0928 08:57:34.121405 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:34.121413 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:34.121460 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:34.121765 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:34.121766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:34.121768 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:57:34.121804 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:34.121806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:34.123054 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:34.123055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:34.123995 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:34.123997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:34.124887 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:34.124888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:34.124959 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:34.124963 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:57:34.125009 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 08:57:34.125048 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":82,"threads_started":1}
I20260928 08:57:34.125416 11913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:34.120805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53832 (local address 127.23.167.195:38407)
0928 08:57:34.121078 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:34.121083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:34.121633 (+ 550us) server_negotiation.cc:407] Connection header received
0928 08:57:34.121677 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:34.121679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:34.121693 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:34.121760 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:57:34.121861 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:34.122968 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:34.124067 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:34.124700 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:34.124770 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:34.124850 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:34.125291 (+ 441us) server_negotiation.cc:299] Negotiation successful
0928 08:57:34.125335 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":89,"threads_started":1}
I20260928 08:57:34.125644 11912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:34.120943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:36805 (local address 127.0.0.1:41880)
0928 08:57:34.121530 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:34.121537 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:34.121588 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:34.121831 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:34.121832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:34.121834 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:57:34.121881 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:34.121882 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:34.123092 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:34.123093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:34.124201 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:34.124204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:34.125432 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:34.125433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:34.125493 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:34.125496 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:57:34.125547 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 08:57:34.125584 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":60,"threads_started":1}
I20260928 08:57:34.125633 11706 tablet_service.cc:1467] Processing CreateTablet for tablet 8f348ec1660443e5b8cf39a2b1a5a2cb (DEFAULT_TABLE table=test-table [id=96dd62cb1d4d487488566b84920011c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:57:34.125896 11916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:34.121148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41880 (local address 127.23.167.193:36805)
0928 08:57:34.121300 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:34.121304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:34.121636 (+ 332us) server_negotiation.cc:407] Connection header received
0928 08:57:34.121679 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:34.121682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:34.121697 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:34.121765 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:57:34.121954 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:34.123036 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:34.124281 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:34.125246 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:34.125305 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:34.125388 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:34.125775 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 08:57:34.125819 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20260928 08:57:34.126014 11706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f348ec1660443e5b8cf39a2b1a5a2cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:34.126121 11837 tablet_service.cc:1467] Processing CreateTablet for tablet 8f348ec1660443e5b8cf39a2b1a5a2cb (DEFAULT_TABLE table=test-table [id=96dd62cb1d4d487488566b84920011c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:57:34.126447 11837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f348ec1660443e5b8cf39a2b1a5a2cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:34.126463 11575 tablet_service.cc:1467] Processing CreateTablet for tablet 8f348ec1660443e5b8cf39a2b1a5a2cb (DEFAULT_TABLE table=test-table [id=96dd62cb1d4d487488566b84920011c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:57:34.126771 11575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f348ec1660443e5b8cf39a2b1a5a2cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:34.128643 11917 tablet_bootstrap.cc:492] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b89d2c1da8624b0cbb3aefc5c0e5882f: Bootstrap starting.
I20260928 08:57:34.129302 11917 tablet_bootstrap.cc:654] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b89d2c1da8624b0cbb3aefc5c0e5882f: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:34.129566 11917 log.cc:824] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b89d2c1da8624b0cbb3aefc5c0e5882f: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:34.129967 11918 tablet_bootstrap.cc:492] T 8f348ec1660443e5b8cf39a2b1a5a2cb P fa92de1c9dfb4b3797fe7aa315d8673e: Bootstrap starting.
I20260928 08:57:34.130554 11918 tablet_bootstrap.cc:654] T 8f348ec1660443e5b8cf39a2b1a5a2cb P fa92de1c9dfb4b3797fe7aa315d8673e: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:34.130580 11917 tablet_bootstrap.cc:492] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b89d2c1da8624b0cbb3aefc5c0e5882f: No bootstrap required, opened a new log
I20260928 08:57:34.130636 11917 ts_tablet_manager.cc:1397] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b89d2c1da8624b0cbb3aefc5c0e5882f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:57:34.130825 11918 log.cc:824] T 8f348ec1660443e5b8cf39a2b1a5a2cb P fa92de1c9dfb4b3797fe7aa315d8673e: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:34.131155 11919 tablet_bootstrap.cc:492] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b84c9ebe6d1c4cd1aa5e899de71986dc: Bootstrap starting.
I20260928 08:57:34.131707 11919 tablet_bootstrap.cc:654] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b84c9ebe6d1c4cd1aa5e899de71986dc: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:34.131968 11919 log.cc:824] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b84c9ebe6d1c4cd1aa5e899de71986dc: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:34.132411 11918 tablet_bootstrap.cc:492] T 8f348ec1660443e5b8cf39a2b1a5a2cb P fa92de1c9dfb4b3797fe7aa315d8673e: No bootstrap required, opened a new log
I20260928 08:57:34.132498 11918 ts_tablet_manager.cc:1397] T 8f348ec1660443e5b8cf39a2b1a5a2cb P fa92de1c9dfb4b3797fe7aa315d8673e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 08:57:34.132695 11919 tablet_bootstrap.cc:492] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b84c9ebe6d1c4cd1aa5e899de71986dc: No bootstrap required, opened a new log
I20260928 08:57:34.132754 11919 ts_tablet_manager.cc:1397] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b84c9ebe6d1c4cd1aa5e899de71986dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:57:34.132762 11917 raft_consensus.cc:348] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b89d2c1da8624b0cbb3aefc5c0e5882f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b89d2c1da8624b0cbb3aefc5c0e5882f" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 41435 } } peers { permanent_uuid: "fa92de1c9dfb4b3797fe7aa315d8673e" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 38407 } } peers { permanent_uuid: "b84c9ebe6d1c4cd1aa5e899de71986dc" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 36805 } }
I20260928 08:57:34.132967 11917 raft_consensus.cc:374] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b89d2c1da8624b0cbb3aefc5c0e5882f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:34.133047 11917 raft_consensus.cc:729] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b89d2c1da8624b0cbb3aefc5c0e5882f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89d2c1da8624b0cbb3aefc5c0e5882f, State: Initialized, Role: FOLLOWER
I20260928 08:57:34.133206 11917 consensus_queue.cc:260] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b89d2c1da8624b0cbb3aefc5c0e5882f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b89d2c1da8624b0cbb3aefc5c0e5882f" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 41435 } } peers { permanent_uuid: "fa92de1c9dfb4b3797fe7aa315d8673e" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 38407 } } peers { permanent_uuid: "b84c9ebe6d1c4cd1aa5e899de71986dc" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 36805 } }
I20260928 08:57:34.133456 11917 ts_tablet_manager.cc:1428] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b89d2c1da8624b0cbb3aefc5c0e5882f: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260928 08:57:34.133594 11772 heartbeater.cc:503] Master 127.23.167.254:46343 was elected leader, sending a full tablet report...
I20260928 08:57:34.135253 11919 raft_consensus.cc:348] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b84c9ebe6d1c4cd1aa5e899de71986dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b89d2c1da8624b0cbb3aefc5c0e5882f" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 41435 } } peers { permanent_uuid: "fa92de1c9dfb4b3797fe7aa315d8673e" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 38407 } } peers { permanent_uuid: "b84c9ebe6d1c4cd1aa5e899de71986dc" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 36805 } }
I20260928 08:57:34.135432 11919 raft_consensus.cc:374] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b84c9ebe6d1c4cd1aa5e899de71986dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:34.135475 11919 raft_consensus.cc:729] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b84c9ebe6d1c4cd1aa5e899de71986dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b84c9ebe6d1c4cd1aa5e899de71986dc, State: Initialized, Role: FOLLOWER
I20260928 08:57:34.135530 11918 raft_consensus.cc:348] T 8f348ec1660443e5b8cf39a2b1a5a2cb P fa92de1c9dfb4b3797fe7aa315d8673e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b89d2c1da8624b0cbb3aefc5c0e5882f" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 41435 } } peers { permanent_uuid: "fa92de1c9dfb4b3797fe7aa315d8673e" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 38407 } } peers { permanent_uuid: "b84c9ebe6d1c4cd1aa5e899de71986dc" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 36805 } }
I20260928 08:57:34.135599 11919 consensus_queue.cc:260] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b84c9ebe6d1c4cd1aa5e899de71986dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b89d2c1da8624b0cbb3aefc5c0e5882f" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 41435 } } peers { permanent_uuid: "fa92de1c9dfb4b3797fe7aa315d8673e" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 38407 } } peers { permanent_uuid: "b84c9ebe6d1c4cd1aa5e899de71986dc" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 36805 } }
I20260928 08:57:34.135672 11918 raft_consensus.cc:374] T 8f348ec1660443e5b8cf39a2b1a5a2cb P fa92de1c9dfb4b3797fe7aa315d8673e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:34.135711 11918 raft_consensus.cc:729] T 8f348ec1660443e5b8cf39a2b1a5a2cb P fa92de1c9dfb4b3797fe7aa315d8673e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa92de1c9dfb4b3797fe7aa315d8673e, State: Initialized, Role: FOLLOWER
I20260928 08:57:34.135820 11918 consensus_queue.cc:260] T 8f348ec1660443e5b8cf39a2b1a5a2cb P fa92de1c9dfb4b3797fe7aa315d8673e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b89d2c1da8624b0cbb3aefc5c0e5882f" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 41435 } } peers { permanent_uuid: "fa92de1c9dfb4b3797fe7aa315d8673e" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 38407 } } peers { permanent_uuid: "b84c9ebe6d1c4cd1aa5e899de71986dc" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 36805 } }
I20260928 08:57:34.135878 11919 ts_tablet_manager.cc:1428] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b84c9ebe6d1c4cd1aa5e899de71986dc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:57:34.135897 11641 heartbeater.cc:503] Master 127.23.167.254:46343 was elected leader, sending a full tablet report...
I20260928 08:57:34.136076 11918 ts_tablet_manager.cc:1428] T 8f348ec1660443e5b8cf39a2b1a5a2cb P fa92de1c9dfb4b3797fe7aa315d8673e: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260928 08:57:34.136119 11903 heartbeater.cc:503] Master 127.23.167.254:46343 was elected leader, sending a full tablet report...
I20260928 08:57:34.139874 11924 raft_consensus.cc:482] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b84c9ebe6d1c4cd1aa5e899de71986dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:57:34.139971 11924 raft_consensus.cc:504] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b84c9ebe6d1c4cd1aa5e899de71986dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b89d2c1da8624b0cbb3aefc5c0e5882f" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 41435 } } peers { permanent_uuid: "fa92de1c9dfb4b3797fe7aa315d8673e" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 38407 } } peers { permanent_uuid: "b84c9ebe6d1c4cd1aa5e899de71986dc" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 36805 } }
I20260928 08:57:34.140306 11924 leader_election.cc:288] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b84c9ebe6d1c4cd1aa5e899de71986dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b89d2c1da8624b0cbb3aefc5c0e5882f (127.23.167.194:41435), fa92de1c9dfb4b3797fe7aa315d8673e (127.23.167.195:38407)
I20260928 08:57:34.143000 11643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:34.140463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.194:41435 (local address 127.23.167.193:58163)
0928 08:57:34.140543 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:34.140557 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:34.140600 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:34.140726 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:34.140728 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:34.140729 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:57:34.140806 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:34.140808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:34.141464 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:34.141465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:34.142221 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:34.142223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:34.142742 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:34.142743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:34.142832 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:34.142835 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:57:34.142877 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 08:57:34.142920 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 08:57:34.142990 11915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:34.140541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:58163 (local address 127.23.167.194:41435)
0928 08:57:34.140582 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:34.140584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:34.140592 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:57:34.140664 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:34.140665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:34.140674 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:34.140719 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:57:34.140859 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:34.141391 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:34.142307 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:34.142675 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:34.142734 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:34.142742 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:34.142889 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 08:57:34.142928 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 08:57:34.143234 11913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:34.140787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.167.193:38433 (local address 127.23.167.195:38407)
0928 08:57:34.140844 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:34.140846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:34.140853 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:57:34.140873 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:34.140874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:34.140880 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:34.140931 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:57:34.141086 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:34.141570 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:34.142461 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:34.142840 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:34.142906 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:34.142924 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:34.143122 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 08:57:34.143167 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 08:57:34.143345 11726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f348ec1660443e5b8cf39a2b1a5a2cb" candidate_uuid: "b84c9ebe6d1c4cd1aa5e899de71986dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89d2c1da8624b0cbb3aefc5c0e5882f" is_pre_election: true
I20260928 08:57:34.143429 11926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:34.140495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.195:38407 (local address 127.23.167.193:38433)
0928 08:57:34.140714 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:34.140723 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:34.140762 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:34.140940 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:34.140940 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:34.140942 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:57:34.141010 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:34.141011 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:34.141648 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:34.141649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:34.142383 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:34.142385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:34.142959 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:34.142960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:34.143036 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:34.143039 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:57:34.143318 (+ 279us) client_negotiation.cc:240] Negotiation successful
0928 08:57:34.143361 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":100,"threads_started":1}
I20260928 08:57:34.143509 11726 raft_consensus.cc:2454] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b89d2c1da8624b0cbb3aefc5c0e5882f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b84c9ebe6d1c4cd1aa5e899de71986dc in term 0.
I20260928 08:57:34.143724 11531 leader_election.cc:302] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b84c9ebe6d1c4cd1aa5e899de71986dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b84c9ebe6d1c4cd1aa5e899de71986dc, b89d2c1da8624b0cbb3aefc5c0e5882f; no voters:
I20260928 08:57:34.143841 11857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f348ec1660443e5b8cf39a2b1a5a2cb" candidate_uuid: "b84c9ebe6d1c4cd1aa5e899de71986dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa92de1c9dfb4b3797fe7aa315d8673e" is_pre_election: true
I20260928 08:57:34.143901 11924 raft_consensus.cc:2781] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b84c9ebe6d1c4cd1aa5e899de71986dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:57:34.143944 11924 raft_consensus.cc:482] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b84c9ebe6d1c4cd1aa5e899de71986dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:57:34.143957 11924 raft_consensus.cc:3037] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b84c9ebe6d1c4cd1aa5e899de71986dc [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:34.143977 11857 raft_consensus.cc:2454] T 8f348ec1660443e5b8cf39a2b1a5a2cb P fa92de1c9dfb4b3797fe7aa315d8673e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b84c9ebe6d1c4cd1aa5e899de71986dc in term 0.
I20260928 08:57:34.144683 11924 raft_consensus.cc:504] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b84c9ebe6d1c4cd1aa5e899de71986dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b89d2c1da8624b0cbb3aefc5c0e5882f" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 41435 } } peers { permanent_uuid: "fa92de1c9dfb4b3797fe7aa315d8673e" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 38407 } } peers { permanent_uuid: "b84c9ebe6d1c4cd1aa5e899de71986dc" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 36805 } }
I20260928 08:57:34.144806 11924 leader_election.cc:288] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b84c9ebe6d1c4cd1aa5e899de71986dc [CANDIDATE]: Term 1 election: Requested vote from peers b89d2c1da8624b0cbb3aefc5c0e5882f (127.23.167.194:41435), fa92de1c9dfb4b3797fe7aa315d8673e (127.23.167.195:38407)
I20260928 08:57:34.144948 11726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f348ec1660443e5b8cf39a2b1a5a2cb" candidate_uuid: "b84c9ebe6d1c4cd1aa5e899de71986dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89d2c1da8624b0cbb3aefc5c0e5882f"
I20260928 08:57:34.144982 11857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f348ec1660443e5b8cf39a2b1a5a2cb" candidate_uuid: "b84c9ebe6d1c4cd1aa5e899de71986dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa92de1c9dfb4b3797fe7aa315d8673e"
I20260928 08:57:34.145026 11726 raft_consensus.cc:3037] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b89d2c1da8624b0cbb3aefc5c0e5882f [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:34.145085 11857 raft_consensus.cc:3037] T 8f348ec1660443e5b8cf39a2b1a5a2cb P fa92de1c9dfb4b3797fe7aa315d8673e [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:34.145849 11857 raft_consensus.cc:2454] T 8f348ec1660443e5b8cf39a2b1a5a2cb P fa92de1c9dfb4b3797fe7aa315d8673e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b84c9ebe6d1c4cd1aa5e899de71986dc in term 1.
I20260928 08:57:34.145853 11726 raft_consensus.cc:2454] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b89d2c1da8624b0cbb3aefc5c0e5882f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b84c9ebe6d1c4cd1aa5e899de71986dc in term 1.
I20260928 08:57:34.145982 11531 leader_election.cc:302] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b84c9ebe6d1c4cd1aa5e899de71986dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b84c9ebe6d1c4cd1aa5e899de71986dc, b89d2c1da8624b0cbb3aefc5c0e5882f; no voters:
I20260928 08:57:34.146081 11924 raft_consensus.cc:2781] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b84c9ebe6d1c4cd1aa5e899de71986dc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:57:34.146241 11924 raft_consensus.cc:686] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b84c9ebe6d1c4cd1aa5e899de71986dc [term 1 LEADER]: Becoming Leader. State: Replica: b84c9ebe6d1c4cd1aa5e899de71986dc, State: Running, Role: LEADER
I20260928 08:57:34.146322 11924 consensus_queue.cc:237] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b84c9ebe6d1c4cd1aa5e899de71986dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b89d2c1da8624b0cbb3aefc5c0e5882f" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 41435 } } peers { permanent_uuid: "fa92de1c9dfb4b3797fe7aa315d8673e" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 38407 } } peers { permanent_uuid: "b84c9ebe6d1c4cd1aa5e899de71986dc" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 36805 } }
I20260928 08:57:34.147109 11452 catalog_manager.cc:5495] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b84c9ebe6d1c4cd1aa5e899de71986dc reported cstate change: term changed from 0 to 1, leader changed from <none> to b84c9ebe6d1c4cd1aa5e899de71986dc (127.23.167.193). New cstate: current_term: 1 leader_uuid: "b84c9ebe6d1c4cd1aa5e899de71986dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b89d2c1da8624b0cbb3aefc5c0e5882f" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 41435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa92de1c9dfb4b3797fe7aa315d8673e" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 38407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b84c9ebe6d1c4cd1aa5e899de71986dc" member_type: VOTER last_known_addr { host: "127.23.167.193" port: 36805 } health_report { overall_health: HEALTHY } } }
I20260928 08:57:34.157032 11916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:34.154661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41882 (local address 127.23.167.193:36805)
0928 08:57:34.154725 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:34.154727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:34.154735 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:57:34.154772 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:34.154773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:34.154793 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:34.154853 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:57:34.154985 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:34.155492 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:34.156258 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:34.156458 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:34.156514 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:34.156678 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:57:34.156793 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:57:34.156868 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:34.156908 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 08:57:34.156931 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 08:57:34.159017 11857 raft_consensus.cc:1261] T 8f348ec1660443e5b8cf39a2b1a5a2cb P fa92de1c9dfb4b3797fe7aa315d8673e [term 1 FOLLOWER]: Refusing update from remote peer b84c9ebe6d1c4cd1aa5e899de71986dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:57:34.159018 11726 raft_consensus.cc:1261] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b89d2c1da8624b0cbb3aefc5c0e5882f [term 1 FOLLOWER]: Refusing update from remote peer b84c9ebe6d1c4cd1aa5e899de71986dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:57:34.159401 11924 consensus_queue.cc:1035] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b84c9ebe6d1c4cd1aa5e899de71986dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89d2c1da8624b0cbb3aefc5c0e5882f" member_type: VOTER last_known_addr { host: "127.23.167.194" port: 41435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:57:34.159544 11927 consensus_queue.cc:1035] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b84c9ebe6d1c4cd1aa5e899de71986dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa92de1c9dfb4b3797fe7aa315d8673e" member_type: VOTER last_known_addr { host: "127.23.167.195" port: 38407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:57:34.162340 11930 mvcc.cc:205] Tried to move back new op lower bound from 7334239658632454144 to 7334239658583957504. Current Snapshot: MvccSnapshot[applied={T|T < 7334239658632454144}]
I20260928 08:57:34.162482 11931 mvcc.cc:205] Tried to move back new op lower bound from 7334239658632454144 to 7334239658583957504. Current Snapshot: MvccSnapshot[applied={T|T < 7334239658632454144}]
I20260928 08:57:34.162529 11932 mvcc.cc:205] Tried to move back new op lower bound from 7334239658632454144 to 7334239658583957504. Current Snapshot: MvccSnapshot[applied={T|T < 7334239658632454144}]
I20260928 08:57:34.164884 11452 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:57:34.164966 11452 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:57:34.165688 11452 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3db08cbf9644e73917eb33b6d034f27: Sending DeleteTablet for 3 replicas of tablet 8f348ec1660443e5b8cf39a2b1a5a2cb
I20260928 08:57:34.166018 11575 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f348ec1660443e5b8cf39a2b1a5a2cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:57:34 UTC) from {username='slave'} at 127.0.0.1:41880
I20260928 08:57:34.166038 11706 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f348ec1660443e5b8cf39a2b1a5a2cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:57:34 UTC) from {username='slave'} at 127.0.0.1:60350
I20260928 08:57:34.166249 11837 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f348ec1660443e5b8cf39a2b1a5a2cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:57:34 UTC) from {username='slave'} at 127.0.0.1:53832
I20260928 08:57:34.166458 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 11513
I20260928 08:57:34.166481 11942 tablet_replica.cc:330] T 8f348ec1660443e5b8cf39a2b1a5a2cb P fa92de1c9dfb4b3797fe7aa315d8673e: stopping tablet replica
I20260928 08:57:34.166484 11940 tablet_replica.cc:330] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b84c9ebe6d1c4cd1aa5e899de71986dc: stopping tablet replica
I20260928 08:57:34.166652 11942 raft_consensus.cc:2229] T 8f348ec1660443e5b8cf39a2b1a5a2cb P fa92de1c9dfb4b3797fe7aa315d8673e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:57:34.166798 11942 raft_consensus.cc:2258] T 8f348ec1660443e5b8cf39a2b1a5a2cb P fa92de1c9dfb4b3797fe7aa315d8673e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:57:34.167414 11942 ts_tablet_manager.cc:1905] T 8f348ec1660443e5b8cf39a2b1a5a2cb P fa92de1c9dfb4b3797fe7aa315d8673e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:57:34.168041 11941 tablet_replica.cc:330] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b89d2c1da8624b0cbb3aefc5c0e5882f: stopping tablet replica
I20260928 08:57:34.168151 11941 raft_consensus.cc:2229] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b89d2c1da8624b0cbb3aefc5c0e5882f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:57:34.168257 11941 raft_consensus.cc:2258] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b89d2c1da8624b0cbb3aefc5c0e5882f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:57:34.168810 11941 ts_tablet_manager.cc:1905] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b89d2c1da8624b0cbb3aefc5c0e5882f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:57:34.169207 11942 ts_tablet_manager.cc:1918] T 8f348ec1660443e5b8cf39a2b1a5a2cb P fa92de1c9dfb4b3797fe7aa315d8673e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:57:34.169268 11942 log.cc:1196] T 8f348ec1660443e5b8cf39a2b1a5a2cb P fa92de1c9dfb4b3797fe7aa315d8673e: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-2/wal/wals/8f348ec1660443e5b8cf39a2b1a5a2cb
I20260928 08:57:34.169426 11942 ts_tablet_manager.cc:1939] T 8f348ec1660443e5b8cf39a2b1a5a2cb P fa92de1c9dfb4b3797fe7aa315d8673e: Deleting consensus metadata
I20260928 08:57:34.169936 11439 catalog_manager.cc:4841] TS fa92de1c9dfb4b3797fe7aa315d8673e (127.23.167.195:38407): tablet 8f348ec1660443e5b8cf39a2b1a5a2cb (table test-table [id=96dd62cb1d4d487488566b84920011c5]) successfully deleted
I20260928 08:57:34.170686 11941 ts_tablet_manager.cc:1918] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b89d2c1da8624b0cbb3aefc5c0e5882f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:57:34.170763 11941 log.cc:1196] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b89d2c1da8624b0cbb3aefc5c0e5882f: Deleting WAL directory at /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585589999956-24223-0/minicluster-data/ts-1/wal/wals/8f348ec1660443e5b8cf39a2b1a5a2cb
I20260928 08:57:34.170997 11941 ts_tablet_manager.cc:1939] T 8f348ec1660443e5b8cf39a2b1a5a2cb P b89d2c1da8624b0cbb3aefc5c0e5882f: Deleting consensus metadata
I20260928 08:57:34.171452 11440 catalog_manager.cc:4841] TS b89d2c1da8624b0cbb3aefc5c0e5882f (127.23.167.194:41435): tablet 8f348ec1660443e5b8cf39a2b1a5a2cb (table test-table [id=96dd62cb1d4d487488566b84920011c5]) successfully deleted
W20260928 08:57:34.172505 11440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.167.193:36805 (error 108)
I20260928 08:57:34.173027 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 11644
I20260928 08:57:34.173045 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:34.172843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.193:36805 (local address 127.0.0.1:41888)
0928 08:57:34.172923 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:34.172977 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.193:36805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 08:57:34.173900 11440 catalog_manager.cc:4584] TS b84c9ebe6d1c4cd1aa5e899de71986dc (127.23.167.193:36805): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8f348ec1660443e5b8cf39a2b1a5a2cb: Network error: Client connection negotiation failed: client connection to 127.23.167.193:36805: connect: Connection refused (error 111)
I20260928 08:57:34.177902 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 11775
I20260928 08:57:34.181668 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 11422
2026-09-28T08:57:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (636 ms)
[----------] 6 tests from EncryptionPolicyTest (7793 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:57:34 dist-test-slave-pbjp krb5kdc[11946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:57:34 dist-test-slave-pbjp krb5kdc[11946](info): set up 2 sockets
Sep 28 08:57:34 dist-test-slave-pbjp krb5kdc[11946](info): commencing operation
krb5kdc: starting...
W20260928 08:57:35.132820 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:57:35 dist-test-slave-pbjp krb5kdc[11946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:57:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:57:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:57:35.244017 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45441
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:35.328102 11959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:35.328315 11959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:35.328337 11959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:35.329559 11959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:57:35.329602 11959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:35.329615 11959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:35.329635 11959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:57:35.329646 11959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:57:35.331074 11959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45441
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:35.331264 11959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:35.331441 11959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:35.336205 11967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:35.336232 11965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:35.336232 11964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:35.336411 11959 server_base.cc:1034] running on GCE node
I20260928 08:57:35.336632 11959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:35.336891 11959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:35.338029 11959 hybrid_clock.cc:648] HybridClock initialized: now 1790585855338007 us; error 35 us; skew 500 ppm
Sep 28 08:57:35 dist-test-slave-pbjp krb5kdc[11946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:35.340382 11959 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:57:35.340849 11959 webserver.cc:460] Webserver started at http://127.0.0.1:39741/ using document root <none> and password file <none>
I20260928 08:57:35.341046 11959 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:35.341128 11959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:35.341259 11959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:35.342283 11959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "9f1459cafad84546878c104c0e92f041"
format_stamp: "Formatted at 2026-09-28 08:57:35 on dist-test-slave-pbjp"
server_key: "b37393e707b54ef51c46a48e9f5398cd"
server_key_iv: "ce098f69327271c68403cbd735aeccca"
server_key_version: "encryptionkey@0"
I20260928 08:57:35.342619 11959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "9f1459cafad84546878c104c0e92f041"
format_stamp: "Formatted at 2026-09-28 08:57:35 on dist-test-slave-pbjp"
server_key: "b37393e707b54ef51c46a48e9f5398cd"
server_key_iv: "ce098f69327271c68403cbd735aeccca"
server_key_version: "encryptionkey@0"
I20260928 08:57:35.343819 11959 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:57:35.344445 11974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:35.344605 11959 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:57:35.344687 11959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "9f1459cafad84546878c104c0e92f041"
format_stamp: "Formatted at 2026-09-28 08:57:35 on dist-test-slave-pbjp"
server_key: "b37393e707b54ef51c46a48e9f5398cd"
server_key_iv: "ce098f69327271c68403cbd735aeccca"
server_key_version: "encryptionkey@0"
I20260928 08:57:35.344753 11959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:35.360927 11959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:35.362800 11959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:35.362941 11959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:35.366750 11959 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45441
I20260928 08:57:35.366786 12026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45441 every 8 connection(s)
I20260928 08:57:35.367131 11959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:57:35.367981 12027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:35.369213 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 11959
I20260928 08:57:35.369308 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:57:35.369468 24223 external_mini_cluster.cc:1428] Setting key 9959b9cd2d9f64df366c8ea4b579b2e7
I20260928 08:57:35.370608 12027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f1459cafad84546878c104c0e92f041: Bootstrap starting.
I20260928 08:57:35.371263 12027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f1459cafad84546878c104c0e92f041: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:35.371546 12027 log.cc:824] T 00000000000000000000000000000000 P 9f1459cafad84546878c104c0e92f041: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:35.372248 12027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f1459cafad84546878c104c0e92f041: No bootstrap required, opened a new log
I20260928 08:57:35.374382 12027 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f1459cafad84546878c104c0e92f041 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1459cafad84546878c104c0e92f041" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45441 } }
I20260928 08:57:35.374557 12027 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f1459cafad84546878c104c0e92f041 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:35.374590 12027 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f1459cafad84546878c104c0e92f041 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f1459cafad84546878c104c0e92f041, State: Initialized, Role: FOLLOWER
I20260928 08:57:35.374718 12027 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f1459cafad84546878c104c0e92f041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1459cafad84546878c104c0e92f041" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45441 } }
I20260928 08:57:35.374806 12027 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f1459cafad84546878c104c0e92f041 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:57:35.374845 12027 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f1459cafad84546878c104c0e92f041 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:57:35.374895 12027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f1459cafad84546878c104c0e92f041 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:35.375402 12027 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f1459cafad84546878c104c0e92f041 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1459cafad84546878c104c0e92f041" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45441 } }
I20260928 08:57:35.375486 12027 leader_election.cc:302] T 00000000000000000000000000000000 P 9f1459cafad84546878c104c0e92f041 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f1459cafad84546878c104c0e92f041; no voters:
I20260928 08:57:35.375635 12027 leader_election.cc:288] T 00000000000000000000000000000000 P 9f1459cafad84546878c104c0e92f041 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:57:35.375692 12032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f1459cafad84546878c104c0e92f041 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:57:35.375854 12032 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f1459cafad84546878c104c0e92f041 [term 1 LEADER]: Becoming Leader. State: Replica: 9f1459cafad84546878c104c0e92f041, State: Running, Role: LEADER
I20260928 08:57:35.375872 12027 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f1459cafad84546878c104c0e92f041 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:57:35.375945 12032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f1459cafad84546878c104c0e92f041 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1459cafad84546878c104c0e92f041" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45441 } }
I20260928 08:57:35.376281 12032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f1459cafad84546878c104c0e92f041 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f1459cafad84546878c104c0e92f041. Latest consensus state: current_term: 1 leader_uuid: "9f1459cafad84546878c104c0e92f041" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1459cafad84546878c104c0e92f041" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45441 } } }
I20260928 08:57:35.376289 12034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f1459cafad84546878c104c0e92f041 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f1459cafad84546878c104c0e92f041" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1459cafad84546878c104c0e92f041" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45441 } } }
I20260928 08:57:35.376354 12032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f1459cafad84546878c104c0e92f041 [sys.catalog]: This master's current role is: LEADER
Sep 28 08:57:35 dist-test-slave-pbjp krb5kdc[11946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 08:57:35.376381 12034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f1459cafad84546878c104c0e92f041 [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:35.377161 12039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:57:35.377624 12039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:57:35.379343 12039 catalog_manager.cc:1347] Generated new cluster ID: 5ee588a988244496ac3a2e84ceb08f71
I20260928 08:57:35.379384 12039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:57:35.380513 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:35.370023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58393 (local address 127.0.0.1:45441)
0928 08:57:35.370214 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:35.370225 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:35.370262 (+ 37us) server_negotiation.cc:407] Connection header received
0928 08:57:35.370758 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:35.370767 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:35.370827 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:35.370986 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:35.371528 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:35.372802 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:35.373620 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:35.374267 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:35.374379 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:35.376895 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:35.376900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:35.376900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:35.376913 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:35.378940 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:35.379312 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:35.379327 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:35.379327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:35.379437 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:35.379662 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:35.379663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:35.379663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:35.379969 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:35.380122 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:35.380207 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 08:57:35.380307 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":80,"threads_started":1}
I20260928 08:57:35.381441 24223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:57:35.391674 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:35.382875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57397 (local address 127.0.0.1:45441)
0928 08:57:35.382905 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:35.382907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:35.385245 (+ 2338us) server_negotiation.cc:407] Connection header received
0928 08:57:35.385276 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:35.385277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:35.385287 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:35.385349 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:35.385891 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:35.386656 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:35.387595 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:35.387922 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:35.387983 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:35.389216 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:35.389217 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:35.389217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:35.389224 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:35.390618 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:35.390970 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:35.390971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:35.390972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:35.391026 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:35.391202 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:35.391203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:35.391203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:35.391326 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:35.391416 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:35.391521 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 08:57:35.391582 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 08:57:35.398275 12039 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:57:35.398775 12039 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:57:35.401322 12039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f1459cafad84546878c104c0e92f041: Generated new TSK 0
I20260928 08:57:35.401522 12039 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:57:35.410866 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 11959
W20260928 08:57:35.414814 11944 connection.cc:535] client connection to 127.0.0.1:45441 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45441): Connection reset by peer (error 104)
2026-09-28T08:57:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1228 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:57:35 dist-test-slave-pbjp krb5kdc[12058](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:57:35 dist-test-slave-pbjp krb5kdc[12058](info): set up 2 sockets
Sep 28 08:57:35 dist-test-slave-pbjp krb5kdc[12058](info): commencing operation
krb5kdc: starting...
W20260928 08:57:36.364869 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:57:36 dist-test-slave-pbjp krb5kdc[12058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:57:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:57:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:57:36.476126 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46717
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:36.556044 12071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:36.556241 12071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:36.556262 12071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:36.557423 12071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:57:36.557466 12071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:36.557478 12071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:36.557502 12071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:57:36.557519 12071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:57:36.558947 12071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46717
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:36.559160 12071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:36.559376 12071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 08:57:36.563957 12071 server_base.cc:1034] running on GCE node
W20260928 08:57:36.563925 12076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:36.563978 12079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:36.563930 12077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:36.564330 12071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:36.564577 12071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:36.565708 12071 hybrid_clock.cc:648] HybridClock initialized: now 1790585856565694 us; error 33 us; skew 500 ppm
Sep 28 08:57:36 dist-test-slave-pbjp krb5kdc[12058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:36.571270 12071 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:57:36.571681 12071 webserver.cc:460] Webserver started at http://127.0.0.1:41881/ using document root <none> and password file <none>
I20260928 08:57:36.571877 12071 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:36.571923 12071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:36.572052 12071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:36.573134 12071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "280add5604924295a626b921dd06e95c"
format_stamp: "Formatted at 2026-09-28 08:57:36 on dist-test-slave-pbjp"
server_key: "5283f8c842d5a608211000b8bde3f922"
server_key_iv: "4860363e6943fc47219c6565c95c8302"
server_key_version: "encryptionkey@0"
I20260928 08:57:36.573472 12071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "280add5604924295a626b921dd06e95c"
format_stamp: "Formatted at 2026-09-28 08:57:36 on dist-test-slave-pbjp"
server_key: "5283f8c842d5a608211000b8bde3f922"
server_key_iv: "4860363e6943fc47219c6565c95c8302"
server_key_version: "encryptionkey@0"
I20260928 08:57:36.574702 12071 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:57:36.575434 12086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:36.575603 12071 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:57:36.575682 12071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "280add5604924295a626b921dd06e95c"
format_stamp: "Formatted at 2026-09-28 08:57:36 on dist-test-slave-pbjp"
server_key: "5283f8c842d5a608211000b8bde3f922"
server_key_iv: "4860363e6943fc47219c6565c95c8302"
server_key_version: "encryptionkey@0"
I20260928 08:57:36.575749 12071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:36.582659 12071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:36.584429 12071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:36.584558 12071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:36.588126 12071 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46717
I20260928 08:57:36.588165 12138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46717 every 8 connection(s)
I20260928 08:57:36.588460 12071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:57:36.589299 12139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:36.590603 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 12071
I20260928 08:57:36.590705 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:57:36.590876 24223 external_mini_cluster.cc:1428] Setting key 78a9d2e268ff8c220b3a2a9297c9d308
I20260928 08:57:36.591835 12139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 280add5604924295a626b921dd06e95c: Bootstrap starting.
I20260928 08:57:36.592419 12139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 280add5604924295a626b921dd06e95c: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:36.592723 12139 log.cc:824] T 00000000000000000000000000000000 P 280add5604924295a626b921dd06e95c: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:36.593461 12139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 280add5604924295a626b921dd06e95c: No bootstrap required, opened a new log
I20260928 08:57:36.595577 12139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 280add5604924295a626b921dd06e95c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280add5604924295a626b921dd06e95c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46717 } }
I20260928 08:57:36.595728 12139 raft_consensus.cc:374] T 00000000000000000000000000000000 P 280add5604924295a626b921dd06e95c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:36.595763 12139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 280add5604924295a626b921dd06e95c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 280add5604924295a626b921dd06e95c, State: Initialized, Role: FOLLOWER
I20260928 08:57:36.595887 12139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 280add5604924295a626b921dd06e95c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280add5604924295a626b921dd06e95c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46717 } }
I20260928 08:57:36.595960 12139 raft_consensus.cc:388] T 00000000000000000000000000000000 P 280add5604924295a626b921dd06e95c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:57:36.596000 12139 raft_consensus.cc:482] T 00000000000000000000000000000000 P 280add5604924295a626b921dd06e95c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:57:36.596046 12139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 280add5604924295a626b921dd06e95c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:36.596755 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:36.591356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51821 (local address 127.0.0.1:46717)
0928 08:57:36.591557 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:36.591566 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:36.591780 (+ 214us) server_negotiation.cc:407] Connection header received
0928 08:57:36.592310 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:36.592317 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:36.592368 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:36.592489 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:36.592733 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:36.593952 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:36.594725 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:36.595367 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:36.595478 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:36.596108 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:36.596111 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:36.596112 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:36.596308 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:36.596346 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:36.596360 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:36.596446 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:36.596530 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 08:57:36.596560 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":66,"threads_started":1}
I20260928 08:57:36.596773 12139 raft_consensus.cc:504] T 00000000000000000000000000000000 P 280add5604924295a626b921dd06e95c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280add5604924295a626b921dd06e95c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46717 } }
I20260928 08:57:36.596859 12139 leader_election.cc:302] T 00000000000000000000000000000000 P 280add5604924295a626b921dd06e95c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 280add5604924295a626b921dd06e95c; no voters:
I20260928 08:57:36.596993 12139 leader_election.cc:288] T 00000000000000000000000000000000 P 280add5604924295a626b921dd06e95c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:57:36.597054 12144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 280add5604924295a626b921dd06e95c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:57:36.597229 12144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 280add5604924295a626b921dd06e95c [term 1 LEADER]: Becoming Leader. State: Replica: 280add5604924295a626b921dd06e95c, State: Running, Role: LEADER
I20260928 08:57:36.597239 12139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 280add5604924295a626b921dd06e95c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:57:36.597396 12144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 280add5604924295a626b921dd06e95c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280add5604924295a626b921dd06e95c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46717 } }
I20260928 08:57:36.597716 12144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 280add5604924295a626b921dd06e95c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 280add5604924295a626b921dd06e95c. Latest consensus state: current_term: 1 leader_uuid: "280add5604924295a626b921dd06e95c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280add5604924295a626b921dd06e95c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46717 } } }
I20260928 08:57:36.597726 12146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 280add5604924295a626b921dd06e95c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "280add5604924295a626b921dd06e95c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280add5604924295a626b921dd06e95c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46717 } } }
I20260928 08:57:36.597865 12144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 280add5604924295a626b921dd06e95c [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:36.597892 12146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 280add5604924295a626b921dd06e95c [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:36.597952 24223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:57:36.598060 12152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:57:36.598709 12152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:57:36.601172 12152 catalog_manager.cc:1347] Generated new cluster ID: fac24002e93a4bbd862a3f0c4236605a
I20260928 08:57:36.601220 12152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:57:36.603313 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:36.599452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34713 (local address 127.0.0.1:46717)
0928 08:57:36.599484 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:36.599485 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:36.599877 (+ 392us) server_negotiation.cc:407] Connection header received
0928 08:57:36.599949 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:36.599949 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:36.599959 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:36.600017 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:36.600178 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:36.600665 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:36.601755 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:36.602075 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:36.602139 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:36.602847 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:36.602871 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:36.602872 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:36.602876 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:36.602890 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:36.602895 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:36.602924 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:36.603187 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 08:57:36.603219 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 08:57:36.611984 12152 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:57:36.612506 12152 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:57:36.614770 12152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 280add5604924295a626b921dd06e95c: Generated new TSK 0
I20260928 08:57:36.614933 12152 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:57:36.620080 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 12071
W20260928 08:57:36.624176 12056 connection.cc:535] client connection to 127.0.0.1:46717 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46717): Connection reset by peer (error 104)
2026-09-28T08:57:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1209 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:57:36 dist-test-slave-pbjp krb5kdc[12170](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:57:36 dist-test-slave-pbjp krb5kdc[12170](info): set up 2 sockets
Sep 28 08:57:36 dist-test-slave-pbjp krb5kdc[12170](info): commencing operation
krb5kdc: starting...
W20260928 08:57:37.568905 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:57:37 dist-test-slave-pbjp krb5kdc[12170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:57:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:57:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:57:37.683763 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46435
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:37.762269 12183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:37.762478 12183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:37.762517 12183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:37.763662 12183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:57:37.763706 12183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:37.763724 12183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:37.763739 12183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:57:37.763754 12183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:57:37.765237 12183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46435
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:37.765486 12183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:37.765733 12183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:37.770412 12188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:37.770547 12189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:37.770471 12183 server_base.cc:1034] running on GCE node
W20260928 08:57:37.771468 12191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:37.771718 12183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:37.771976 12183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:37.773120 12183 hybrid_clock.cc:648] HybridClock initialized: now 1790585857773098 us; error 32 us; skew 500 ppm
Sep 28 08:57:37 dist-test-slave-pbjp krb5kdc[12170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:37.775429 12183 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:57:37.775827 12183 webserver.cc:460] Webserver started at http://127.0.0.1:37501/ using document root <none> and password file <none>
I20260928 08:57:37.776042 12183 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:37.776087 12183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:37.776206 12183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:37.777273 12183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "6b9e84d2a77e40d28858de805bf54b8c"
format_stamp: "Formatted at 2026-09-28 08:57:37 on dist-test-slave-pbjp"
server_key: "e97f8c575b5db99bcc91a6c14e139f2e"
server_key_iv: "211879809a3962ab0932beaa357b3c42"
server_key_version: "encryptionkey@0"
I20260928 08:57:37.777586 12183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "6b9e84d2a77e40d28858de805bf54b8c"
format_stamp: "Formatted at 2026-09-28 08:57:37 on dist-test-slave-pbjp"
server_key: "e97f8c575b5db99bcc91a6c14e139f2e"
server_key_iv: "211879809a3962ab0932beaa357b3c42"
server_key_version: "encryptionkey@0"
I20260928 08:57:37.778779 12183 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:57:37.779455 12198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:37.779615 12183 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:57:37.779696 12183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "6b9e84d2a77e40d28858de805bf54b8c"
format_stamp: "Formatted at 2026-09-28 08:57:37 on dist-test-slave-pbjp"
server_key: "e97f8c575b5db99bcc91a6c14e139f2e"
server_key_iv: "211879809a3962ab0932beaa357b3c42"
server_key_version: "encryptionkey@0"
I20260928 08:57:37.779750 12183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:37.782968 12183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:37.784823 12183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:37.784941 12183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:37.788621 12183 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46435
I20260928 08:57:37.788653 12250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46435 every 8 connection(s)
I20260928 08:57:37.788954 12183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:57:37.789914 12251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:37.792089 12251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b9e84d2a77e40d28858de805bf54b8c: Bootstrap starting.
I20260928 08:57:37.792627 12251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b9e84d2a77e40d28858de805bf54b8c: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:37.792886 12251 log.cc:824] T 00000000000000000000000000000000 P 6b9e84d2a77e40d28858de805bf54b8c: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:37.793598 12251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b9e84d2a77e40d28858de805bf54b8c: No bootstrap required, opened a new log
I20260928 08:57:37.795202 12251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b9e84d2a77e40d28858de805bf54b8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b9e84d2a77e40d28858de805bf54b8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46435 } }
I20260928 08:57:37.795341 12251 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b9e84d2a77e40d28858de805bf54b8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:37.795383 12251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b9e84d2a77e40d28858de805bf54b8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9e84d2a77e40d28858de805bf54b8c, State: Initialized, Role: FOLLOWER
I20260928 08:57:37.795516 12251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b9e84d2a77e40d28858de805bf54b8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b9e84d2a77e40d28858de805bf54b8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46435 } }
I20260928 08:57:37.795578 12251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b9e84d2a77e40d28858de805bf54b8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:57:37.795607 12251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b9e84d2a77e40d28858de805bf54b8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:57:37.795652 12251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b9e84d2a77e40d28858de805bf54b8c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:37.796160 12251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b9e84d2a77e40d28858de805bf54b8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b9e84d2a77e40d28858de805bf54b8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46435 } }
I20260928 08:57:37.796250 12251 leader_election.cc:302] T 00000000000000000000000000000000 P 6b9e84d2a77e40d28858de805bf54b8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b9e84d2a77e40d28858de805bf54b8c; no voters:
I20260928 08:57:37.796412 12251 leader_election.cc:288] T 00000000000000000000000000000000 P 6b9e84d2a77e40d28858de805bf54b8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:57:37.796452 12254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b9e84d2a77e40d28858de805bf54b8c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:57:37.796604 12254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b9e84d2a77e40d28858de805bf54b8c [term 1 LEADER]: Becoming Leader. State: Replica: 6b9e84d2a77e40d28858de805bf54b8c, State: Running, Role: LEADER
I20260928 08:57:37.796658 12251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b9e84d2a77e40d28858de805bf54b8c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:57:37.796694 12254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b9e84d2a77e40d28858de805bf54b8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b9e84d2a77e40d28858de805bf54b8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46435 } }
I20260928 08:57:37.797036 12256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b9e84d2a77e40d28858de805bf54b8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b9e84d2a77e40d28858de805bf54b8c. Latest consensus state: current_term: 1 leader_uuid: "6b9e84d2a77e40d28858de805bf54b8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b9e84d2a77e40d28858de805bf54b8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46435 } } }
I20260928 08:57:37.797142 12256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b9e84d2a77e40d28858de805bf54b8c [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:37.797176 12255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b9e84d2a77e40d28858de805bf54b8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b9e84d2a77e40d28858de805bf54b8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b9e84d2a77e40d28858de805bf54b8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46435 } } }
I20260928 08:57:37.797242 12255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b9e84d2a77e40d28858de805bf54b8c [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:37.797472 12261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:57:37.797966 12261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:57:37.798085 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 12183
I20260928 08:57:37.798182 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:57:37.798362 24223 external_mini_cluster.cc:1428] Setting key c355a67d717793b1e6bb8ceb6439b504
I20260928 08:57:37.799544 12261 catalog_manager.cc:1347] Generated new cluster ID: 92062b256caf4532aab38b72049e1aec
I20260928 08:57:37.799598 12261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:57:37.801034 12273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:37.798820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37339 (local address 127.0.0.1:46435)
0928 08:57:37.798964 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:37.798973 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:37.799705 (+ 732us) server_negotiation.cc:407] Connection header received
0928 08:57:37.799833 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:37.799838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:37.799861 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:37.799978 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:37.800637 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:37.800640 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:37.800641 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:37.800645 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:37.800682 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:37.800693 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:37.800744 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:37.800807 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 08:57:37.800849 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":69,"threads_started":1}
I20260928 08:57:37.801827 24223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:57:37.806577 12273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:37.805466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34077 (local address 127.0.0.1:46435)
0928 08:57:37.805507 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:37.805509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:37.805641 (+ 132us) server_negotiation.cc:407] Connection header received
0928 08:57:37.805712 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:37.805713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:37.805727 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:37.805774 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:37.806340 (+ 566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:37.806341 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:37.806342 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:37.806346 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:37.806362 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:37.806366 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:37.806413 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:37.806493 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 08:57:37.806508 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 08:57:37.814039 12261 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:57:37.814222 12261 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:57:37.819880 12261 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b9e84d2a77e40d28858de805bf54b8c: Generated new TSK 0
I20260928 08:57:37.820084 12261 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:57:37.821833 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 12183
2026-09-28T08:57:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1201 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:57:37 dist-test-slave-pbjp krb5kdc[12282](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:57:37 dist-test-slave-pbjp krb5kdc[12282](info): set up 2 sockets
Sep 28 08:57:37 dist-test-slave-pbjp krb5kdc[12282](info): commencing operation
krb5kdc: starting...
W20260928 08:57:38.764786 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:57:38 dist-test-slave-pbjp krb5kdc[12282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:57:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:57:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:57:38.875896 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38357
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:38.955732 12295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:38.955955 12295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:38.955986 12295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:38.957186 12295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:57:38.957230 12295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:38.957247 12295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:38.957264 12295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:57:38.957279 12295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:57:38.958781 12295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:38.959033 12295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:38.959247 12295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:38.964063 12300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:38.964213 12301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:38.964063 12303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:38.964308 12295 server_base.cc:1034] running on GCE node
I20260928 08:57:38.964622 12295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:38.964879 12295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:38.966038 12295 hybrid_clock.cc:648] HybridClock initialized: now 1790585858966020 us; error 32 us; skew 500 ppm
Sep 28 08:57:38 dist-test-slave-pbjp krb5kdc[12282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:38.968494 12295 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:57:38.968931 12295 webserver.cc:460] Webserver started at http://127.0.0.1:34171/ using document root <none> and password file <none>
I20260928 08:57:38.969170 12295 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:38.969218 12295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:38.969345 12295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:38.970413 12295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "0ed7bfaf79444bb29a1f5edbd61e2ac9"
format_stamp: "Formatted at 2026-09-28 08:57:38 on dist-test-slave-pbjp"
server_key: "89a10d6c1ec34e8bf09df20201507faa"
server_key_iv: "f17e19daf7d52e2319859380c862da8e"
server_key_version: "encryptionkey@0"
I20260928 08:57:38.970731 12295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "0ed7bfaf79444bb29a1f5edbd61e2ac9"
format_stamp: "Formatted at 2026-09-28 08:57:38 on dist-test-slave-pbjp"
server_key: "89a10d6c1ec34e8bf09df20201507faa"
server_key_iv: "f17e19daf7d52e2319859380c862da8e"
server_key_version: "encryptionkey@0"
I20260928 08:57:38.971999 12295 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 08:57:38.972752 12310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:38.972908 12295 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:57:38.972993 12295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "0ed7bfaf79444bb29a1f5edbd61e2ac9"
format_stamp: "Formatted at 2026-09-28 08:57:38 on dist-test-slave-pbjp"
server_key: "89a10d6c1ec34e8bf09df20201507faa"
server_key_iv: "f17e19daf7d52e2319859380c862da8e"
server_key_version: "encryptionkey@0"
I20260928 08:57:38.973055 12295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:38.988731 12295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:38.990581 12295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:38.990712 12295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:38.994321 12295 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38357
I20260928 08:57:38.994360 12362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38357 every 8 connection(s)
I20260928 08:57:38.994675 12295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:57:38.995477 12363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:38.997699 12363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ed7bfaf79444bb29a1f5edbd61e2ac9: Bootstrap starting.
I20260928 08:57:38.998212 12363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ed7bfaf79444bb29a1f5edbd61e2ac9: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:38.998435 12363 log.cc:824] T 00000000000000000000000000000000 P 0ed7bfaf79444bb29a1f5edbd61e2ac9: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:38.999037 12363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ed7bfaf79444bb29a1f5edbd61e2ac9: No bootstrap required, opened a new log
I20260928 08:57:39.000437 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 12295
I20260928 08:57:39.000530 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:57:39.000543 12363 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ed7bfaf79444bb29a1f5edbd61e2ac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed7bfaf79444bb29a1f5edbd61e2ac9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38357 } }
I20260928 08:57:39.000677 24223 external_mini_cluster.cc:1428] Setting key a38b274634e964a1dab7d8282b7a5580
I20260928 08:57:39.000705 12363 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ed7bfaf79444bb29a1f5edbd61e2ac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:39.000742 12363 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ed7bfaf79444bb29a1f5edbd61e2ac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed7bfaf79444bb29a1f5edbd61e2ac9, State: Initialized, Role: FOLLOWER
I20260928 08:57:39.000890 12363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ed7bfaf79444bb29a1f5edbd61e2ac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed7bfaf79444bb29a1f5edbd61e2ac9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38357 } }
I20260928 08:57:39.000948 12363 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ed7bfaf79444bb29a1f5edbd61e2ac9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:57:39.000972 12363 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ed7bfaf79444bb29a1f5edbd61e2ac9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:57:39.001020 12363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ed7bfaf79444bb29a1f5edbd61e2ac9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:39.001987 12363 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ed7bfaf79444bb29a1f5edbd61e2ac9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed7bfaf79444bb29a1f5edbd61e2ac9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38357 } }
I20260928 08:57:39.002105 12363 leader_election.cc:302] T 00000000000000000000000000000000 P 0ed7bfaf79444bb29a1f5edbd61e2ac9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ed7bfaf79444bb29a1f5edbd61e2ac9; no voters:
I20260928 08:57:39.002265 12363 leader_election.cc:288] T 00000000000000000000000000000000 P 0ed7bfaf79444bb29a1f5edbd61e2ac9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:57:39.002296 12368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ed7bfaf79444bb29a1f5edbd61e2ac9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:57:39.002471 12368 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ed7bfaf79444bb29a1f5edbd61e2ac9 [term 1 LEADER]: Becoming Leader. State: Replica: 0ed7bfaf79444bb29a1f5edbd61e2ac9, State: Running, Role: LEADER
I20260928 08:57:39.002514 12363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ed7bfaf79444bb29a1f5edbd61e2ac9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:57:39.002559 12368 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ed7bfaf79444bb29a1f5edbd61e2ac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed7bfaf79444bb29a1f5edbd61e2ac9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38357 } }
I20260928 08:57:39.002915 12369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ed7bfaf79444bb29a1f5edbd61e2ac9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ed7bfaf79444bb29a1f5edbd61e2ac9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed7bfaf79444bb29a1f5edbd61e2ac9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38357 } } }
I20260928 08:57:39.003024 12369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ed7bfaf79444bb29a1f5edbd61e2ac9 [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:39.003180 12370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ed7bfaf79444bb29a1f5edbd61e2ac9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ed7bfaf79444bb29a1f5edbd61e2ac9. Latest consensus state: current_term: 1 leader_uuid: "0ed7bfaf79444bb29a1f5edbd61e2ac9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed7bfaf79444bb29a1f5edbd61e2ac9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38357 } } }
I20260928 08:57:39.003254 12370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ed7bfaf79444bb29a1f5edbd61e2ac9 [sys.catalog]: This master's current role is: LEADER
W20260928 08:57:39.004276 12383 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ed7bfaf79444bb29a1f5edbd61e2ac9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:57:39.004774 12383 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:57:39.004858 12376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:57:39.005038 12376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:57:39.006702 12376 catalog_manager.cc:1347] Generated new cluster ID: ab2f4672746d4484993d0400b7c6183e
I20260928 08:57:39.006752 12376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 08:57:39 dist-test-slave-pbjp krb5kdc[12282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 08:57:39.011528 12367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:39.001134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41373 (local address 127.0.0.1:38357)
0928 08:57:39.001305 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:39.001315 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:39.001351 (+ 36us) server_negotiation.cc:407] Connection header received
0928 08:57:39.001485 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:39.001491 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:39.001533 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:39.001643 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:39.002449 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:39.003760 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:39.004580 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:39.005233 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:39.005334 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:39.008009 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:39.008013 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:39.008013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:39.008025 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:39.009924 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:39.010304 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:39.010306 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:39.010306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:39.010413 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:39.010633 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:39.010633 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:39.010634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:39.010938 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:39.011072 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:39.011182 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 08:57:39.011309 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
I20260928 08:57:39.012645 12376 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:57:39.012804 24223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:57:39.013271 12376 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:57:39.024658 12367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:39.017369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56749 (local address 127.0.0.1:38357)
0928 08:57:39.017413 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:39.017414 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:39.017544 (+ 130us) server_negotiation.cc:407] Connection header received
0928 08:57:39.017578 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:39.017579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:39.017591 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:39.017647 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:39.018247 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:39.019050 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:39.020215 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:39.020583 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:39.020686 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:39.022021 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:39.022022 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:39.022023 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:39.022030 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:39.023555 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:39.023908 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:39.023908 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:39.023909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:39.023971 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:39.024136 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:39.024137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:39.024137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:39.024266 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:39.024360 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:39.024471 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 08:57:39.024558 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 08:57:39.025137 12376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ed7bfaf79444bb29a1f5edbd61e2ac9: Generated new TSK 0
I20260928 08:57:39.025336 12376 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:57:39.030849 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 12295
2026-09-28T08:57:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1209 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:57:39 dist-test-slave-pbjp krb5kdc[12394](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:57:39 dist-test-slave-pbjp krb5kdc[12394](info): set up 2 sockets
Sep 28 08:57:39 dist-test-slave-pbjp krb5kdc[12394](info): commencing operation
krb5kdc: starting...
W20260928 08:57:39.981134 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:57:40 dist-test-slave-pbjp krb5kdc[12394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:57:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:57:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:57:40.091388 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36795
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:40.170780 12407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:40.171018 12407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:40.171052 12407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:40.172302 12407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:57:40.172363 12407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:40.172384 12407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:40.172399 12407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:57:40.172415 12407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:57:40.174028 12407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:40.174293 12407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:40.174533 12407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 08:57:40.179554 12407 server_base.cc:1034] running on GCE node
W20260928 08:57:40.179474 12412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:40.179476 12415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:40.179467 12413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:40.179953 12407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:40.180271 12407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:40.181419 12407 hybrid_clock.cc:648] HybridClock initialized: now 1790585860181400 us; error 34 us; skew 500 ppm
Sep 28 08:57:40 dist-test-slave-pbjp krb5kdc[12394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:40.183853 12407 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:57:40.184301 12407 webserver.cc:460] Webserver started at http://127.0.0.1:41945/ using document root <none> and password file <none>
I20260928 08:57:40.184522 12407 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:40.184569 12407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:40.184697 12407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:40.185842 12407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "a4c5406ec7154116898f35bc6a6cb338"
format_stamp: "Formatted at 2026-09-28 08:57:40 on dist-test-slave-pbjp"
server_key: "30e3e8ab3f76a984ade61281fc6acf9c"
server_key_iv: "201c686e139e6c5226abdab2d435e042"
server_key_version: "encryptionkey@0"
I20260928 08:57:40.186175 12407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "a4c5406ec7154116898f35bc6a6cb338"
format_stamp: "Formatted at 2026-09-28 08:57:40 on dist-test-slave-pbjp"
server_key: "30e3e8ab3f76a984ade61281fc6acf9c"
server_key_iv: "201c686e139e6c5226abdab2d435e042"
server_key_version: "encryptionkey@0"
I20260928 08:57:40.187428 12407 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 08:57:40.188158 12422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:40.188330 12407 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:57:40.188416 12407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "a4c5406ec7154116898f35bc6a6cb338"
format_stamp: "Formatted at 2026-09-28 08:57:40 on dist-test-slave-pbjp"
server_key: "30e3e8ab3f76a984ade61281fc6acf9c"
server_key_iv: "201c686e139e6c5226abdab2d435e042"
server_key_version: "encryptionkey@0"
I20260928 08:57:40.188476 12407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:40.205772 12407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:40.207603 12407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:40.207744 12407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:40.211812 12407 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36795
I20260928 08:57:40.211867 12474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36795 every 8 connection(s)
I20260928 08:57:40.212205 12407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:57:40.213011 12475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:40.215296 12475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4c5406ec7154116898f35bc6a6cb338: Bootstrap starting.
I20260928 08:57:40.215560 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 12407
I20260928 08:57:40.215684 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:57:40.215899 24223 external_mini_cluster.cc:1428] Setting key 1ac9c281155c83ae87cc38abd640e5b6
I20260928 08:57:40.216010 12475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4c5406ec7154116898f35bc6a6cb338: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:40.216307 12475 log.cc:824] T 00000000000000000000000000000000 P a4c5406ec7154116898f35bc6a6cb338: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:40.217010 12475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4c5406ec7154116898f35bc6a6cb338: No bootstrap required, opened a new log
I20260928 08:57:40.219043 12475 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4c5406ec7154116898f35bc6a6cb338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4c5406ec7154116898f35bc6a6cb338" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36795 } }
I20260928 08:57:40.219216 12475 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4c5406ec7154116898f35bc6a6cb338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:40.219245 12475 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4c5406ec7154116898f35bc6a6cb338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4c5406ec7154116898f35bc6a6cb338, State: Initialized, Role: FOLLOWER
I20260928 08:57:40.219385 12475 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4c5406ec7154116898f35bc6a6cb338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4c5406ec7154116898f35bc6a6cb338" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36795 } }
I20260928 08:57:40.219463 12475 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4c5406ec7154116898f35bc6a6cb338 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:57:40.219538 12475 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4c5406ec7154116898f35bc6a6cb338 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:57:40.219580 12475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4c5406ec7154116898f35bc6a6cb338 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:40.220172 12475 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4c5406ec7154116898f35bc6a6cb338 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4c5406ec7154116898f35bc6a6cb338" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36795 } }
I20260928 08:57:40.220289 12475 leader_election.cc:302] T 00000000000000000000000000000000 P a4c5406ec7154116898f35bc6a6cb338 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4c5406ec7154116898f35bc6a6cb338; no voters:
I20260928 08:57:40.220474 12475 leader_election.cc:288] T 00000000000000000000000000000000 P a4c5406ec7154116898f35bc6a6cb338 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:57:40.220511 12480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4c5406ec7154116898f35bc6a6cb338 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:57:40.220753 12475 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4c5406ec7154116898f35bc6a6cb338 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:57:40.220849 12480 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4c5406ec7154116898f35bc6a6cb338 [term 1 LEADER]: Becoming Leader. State: Replica: a4c5406ec7154116898f35bc6a6cb338, State: Running, Role: LEADER
I20260928 08:57:40.221012 12480 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4c5406ec7154116898f35bc6a6cb338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4c5406ec7154116898f35bc6a6cb338" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36795 } }
I20260928 08:57:40.221436 12481 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4c5406ec7154116898f35bc6a6cb338 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4c5406ec7154116898f35bc6a6cb338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4c5406ec7154116898f35bc6a6cb338" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36795 } } }
I20260928 08:57:40.221522 12481 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4c5406ec7154116898f35bc6a6cb338 [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:40.221668 12482 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4c5406ec7154116898f35bc6a6cb338 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4c5406ec7154116898f35bc6a6cb338. Latest consensus state: current_term: 1 leader_uuid: "a4c5406ec7154116898f35bc6a6cb338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4c5406ec7154116898f35bc6a6cb338" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36795 } } }
I20260928 08:57:40.221704 12478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:40.216439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39509 (local address 127.0.0.1:36795)
0928 08:57:40.216610 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:40.216620 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:40.216644 (+ 24us) server_negotiation.cc:407] Connection header received
0928 08:57:40.217139 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:40.217149 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:40.217213 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:40.217334 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:40.217559 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:40.218429 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:40.219488 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:40.220023 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:40.220114 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:40.220780 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:40.220784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:40.220785 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:40.221000 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:40.221042 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:40.221057 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:40.221408 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:40.221442 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 08:57:40.221514 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
I20260928 08:57:40.221712 12482 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4c5406ec7154116898f35bc6a6cb338 [sys.catalog]: This master's current role is: LEADER
W20260928 08:57:40.222548 12495 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a4c5406ec7154116898f35bc6a6cb338: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:57:40.222631 12495 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:57:40.222643 24223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:57:40.222692 12489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:57:40.222837 12489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:57:40.224249 12489 catalog_manager.cc:1347] Generated new cluster ID: cb13ef53a1be40aa8fc457d9fa20d172
I20260928 08:57:40.224304 12489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:57:40.226845 12478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:40.224079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45605 (local address 127.0.0.1:36795)
0928 08:57:40.224116 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:40.224117 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:40.224325 (+ 208us) server_negotiation.cc:407] Connection header received
0928 08:57:40.224422 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:40.224423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:40.224433 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:40.224478 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:40.224595 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:40.225102 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:40.225800 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:40.226001 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:40.226055 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:40.226643 (+ 588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:40.226645 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:40.226645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:40.226649 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:40.226663 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:40.226668 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:40.226710 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:40.226753 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 08:57:40.226776 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 08:57:40.243310 12489 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:57:40.243881 12489 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:57:40.253125 12489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4c5406ec7154116898f35bc6a6cb338: Generated new TSK 0
I20260928 08:57:40.253355 12489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:57:40.257293 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 12407
2026-09-28T08:57:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1226 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:57:40 dist-test-slave-pbjp krb5kdc[12506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:57:40 dist-test-slave-pbjp krb5kdc[12506](info): set up 2 sockets
Sep 28 08:57:40 dist-test-slave-pbjp krb5kdc[12506](info): commencing operation
krb5kdc: starting...
W20260928 08:57:41.209092 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:57:41 dist-test-slave-pbjp krb5kdc[12506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:57:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:57:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:57:41.326085 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45641
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:41.406291 12519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:41.406497 12519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:41.406527 12519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:41.407675 12519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:57:41.407717 12519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:41.407735 12519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:41.407750 12519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:57:41.407766 12519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:57:41.409291 12519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:41.409535 12519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:41.409737 12519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:41.414233 12524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:41.414276 12525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:41.414443 12519 server_base.cc:1034] running on GCE node
W20260928 08:57:41.414286 12527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:41.414698 12519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:41.414964 12519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:41.416119 12519 hybrid_clock.cc:648] HybridClock initialized: now 1790585861416107 us; error 29 us; skew 500 ppm
Sep 28 08:57:41 dist-test-slave-pbjp krb5kdc[12506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:41.418561 12519 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:57:41.418962 12519 webserver.cc:460] Webserver started at http://127.0.0.1:46389/ using document root <none> and password file <none>
I20260928 08:57:41.419168 12519 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:41.419216 12519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:41.419351 12519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:41.420429 12519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "cd0fe5fd34e54cbea7ba8fb25039486a"
format_stamp: "Formatted at 2026-09-28 08:57:41 on dist-test-slave-pbjp"
server_key: "b650179ea06d1308a670b2b7f1db9668"
server_key_iv: "2b2e5149f46ac79666a088ebac616ff0"
server_key_version: "encryptionkey@0"
I20260928 08:57:41.420753 12519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "cd0fe5fd34e54cbea7ba8fb25039486a"
format_stamp: "Formatted at 2026-09-28 08:57:41 on dist-test-slave-pbjp"
server_key: "b650179ea06d1308a670b2b7f1db9668"
server_key_iv: "2b2e5149f46ac79666a088ebac616ff0"
server_key_version: "encryptionkey@0"
I20260928 08:57:41.422008 12519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 08:57:41.422720 12534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:41.422904 12519 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:57:41.422976 12519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "cd0fe5fd34e54cbea7ba8fb25039486a"
format_stamp: "Formatted at 2026-09-28 08:57:41 on dist-test-slave-pbjp"
server_key: "b650179ea06d1308a670b2b7f1db9668"
server_key_iv: "2b2e5149f46ac79666a088ebac616ff0"
server_key_version: "encryptionkey@0"
I20260928 08:57:41.423036 12519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:41.426290 12519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:41.428017 12519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:41.428144 12519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:41.431764 12519 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45641
I20260928 08:57:41.431802 12586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45641 every 8 connection(s)
I20260928 08:57:41.432097 12519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:57:41.433161 12587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:41.435338 12587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd0fe5fd34e54cbea7ba8fb25039486a: Bootstrap starting.
I20260928 08:57:41.435842 12587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd0fe5fd34e54cbea7ba8fb25039486a: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:41.436070 12587 log.cc:824] T 00000000000000000000000000000000 P cd0fe5fd34e54cbea7ba8fb25039486a: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:41.436666 12587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd0fe5fd34e54cbea7ba8fb25039486a: No bootstrap required, opened a new log
I20260928 08:57:41.438297 12587 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd0fe5fd34e54cbea7ba8fb25039486a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd0fe5fd34e54cbea7ba8fb25039486a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45641 } }
I20260928 08:57:41.438437 12587 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd0fe5fd34e54cbea7ba8fb25039486a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:41.438459 12587 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd0fe5fd34e54cbea7ba8fb25039486a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd0fe5fd34e54cbea7ba8fb25039486a, State: Initialized, Role: FOLLOWER
I20260928 08:57:41.438570 12587 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd0fe5fd34e54cbea7ba8fb25039486a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd0fe5fd34e54cbea7ba8fb25039486a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45641 } }
I20260928 08:57:41.438642 12587 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd0fe5fd34e54cbea7ba8fb25039486a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:57:41.438668 12587 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd0fe5fd34e54cbea7ba8fb25039486a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:57:41.438699 12587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd0fe5fd34e54cbea7ba8fb25039486a [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:41.439160 12587 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd0fe5fd34e54cbea7ba8fb25039486a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd0fe5fd34e54cbea7ba8fb25039486a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45641 } }
I20260928 08:57:41.439239 12587 leader_election.cc:302] T 00000000000000000000000000000000 P cd0fe5fd34e54cbea7ba8fb25039486a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd0fe5fd34e54cbea7ba8fb25039486a; no voters:
I20260928 08:57:41.439368 12587 leader_election.cc:288] T 00000000000000000000000000000000 P cd0fe5fd34e54cbea7ba8fb25039486a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:57:41.439426 12590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd0fe5fd34e54cbea7ba8fb25039486a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:57:41.439585 12590 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd0fe5fd34e54cbea7ba8fb25039486a [term 1 LEADER]: Becoming Leader. State: Replica: cd0fe5fd34e54cbea7ba8fb25039486a, State: Running, Role: LEADER
I20260928 08:57:41.439599 12587 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd0fe5fd34e54cbea7ba8fb25039486a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:57:41.439733 12590 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd0fe5fd34e54cbea7ba8fb25039486a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd0fe5fd34e54cbea7ba8fb25039486a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45641 } }
I20260928 08:57:41.440083 12591 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd0fe5fd34e54cbea7ba8fb25039486a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd0fe5fd34e54cbea7ba8fb25039486a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd0fe5fd34e54cbea7ba8fb25039486a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45641 } } }
I20260928 08:57:41.440080 12592 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd0fe5fd34e54cbea7ba8fb25039486a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd0fe5fd34e54cbea7ba8fb25039486a. Latest consensus state: current_term: 1 leader_uuid: "cd0fe5fd34e54cbea7ba8fb25039486a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd0fe5fd34e54cbea7ba8fb25039486a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45641 } } }
I20260928 08:57:41.440179 12591 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd0fe5fd34e54cbea7ba8fb25039486a [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:41.440184 12592 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd0fe5fd34e54cbea7ba8fb25039486a [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:41.440372 12594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:57:41.440644 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 12519
I20260928 08:57:41.440743 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:57:41.440884 12594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:57:41.440943 24223 external_mini_cluster.cc:1428] Setting key 9c7a3db48a4739228c5a989ddbf1bc42
I20260928 08:57:41.443049 12594 catalog_manager.cc:1347] Generated new cluster ID: 5a71c6eacc114046a1f93c00cf18ded2
I20260928 08:57:41.443147 12594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:57:41.443177 12609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:41.441545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56659 (local address 127.0.0.1:45641)
0928 08:57:41.441684 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:41.441693 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:41.441726 (+ 33us) server_negotiation.cc:407] Connection header received
0928 08:57:41.441843 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:41.441850 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:41.441880 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:41.442018 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:41.442675 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:41.442678 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:41.442679 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:41.442684 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:41.442724 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:41.442739 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:41.442789 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:41.442965 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 08:57:41.443003 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":63,"threads_started":1}
I20260928 08:57:41.443953 24223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:57:41.450193 12609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:41.445305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48495 (local address 127.0.0.1:45641)
0928 08:57:41.445334 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:41.445336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:41.449241 (+ 3905us) server_negotiation.cc:407] Connection header received
0928 08:57:41.449307 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:41.449308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:41.449313 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:41.449368 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:41.449916 (+ 548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:41.449917 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:41.449918 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:41.449921 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:41.449936 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:41.449940 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:41.450050 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:41.450110 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 08:57:41.450126 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 08:57:41.455114 12594 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:57:41.455273 12594 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:57:41.463939 12594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd0fe5fd34e54cbea7ba8fb25039486a: Generated new TSK 0
I20260928 08:57:41.464138 12594 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:57:41.466739 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 12519
2026-09-28T08:57:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1209 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:57:41 dist-test-slave-pbjp krb5kdc[12618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:57:41 dist-test-slave-pbjp krb5kdc[12618](info): set up 2 sockets
Sep 28 08:57:41 dist-test-slave-pbjp krb5kdc[12618](info): commencing operation
krb5kdc: starting...
W20260928 08:57:42.420956 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:57:42 dist-test-slave-pbjp krb5kdc[12618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:57:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:57:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:57:42.531899 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39079
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:42.611516 12631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:42.611737 12631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:42.611768 12631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:42.612953 12631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:57:42.612998 12631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:42.613015 12631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:42.613031 12631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:57:42.613045 12631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:57:42.614564 12631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:42.614818 12631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:42.615033 12631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:42.619835 12637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:42.619836 12636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:42.619891 12639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:42.620080 12631 server_base.cc:1034] running on GCE node
I20260928 08:57:42.620302 12631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:42.620580 12631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:42.621753 12631 hybrid_clock.cc:648] HybridClock initialized: now 1790585862621737 us; error 29 us; skew 500 ppm
Sep 28 08:57:42 dist-test-slave-pbjp krb5kdc[12618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585862, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:42.627947 12631 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:57:42.628427 12631 webserver.cc:460] Webserver started at http://127.0.0.1:45151/ using document root <none> and password file <none>
I20260928 08:57:42.628645 12631 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:42.628695 12631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:42.628821 12631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:42.629909 12631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "ac0f6b6a37864ff88931ab601ca0abb2"
format_stamp: "Formatted at 2026-09-28 08:57:42 on dist-test-slave-pbjp"
server_key: "95eb9dcbc10dbdfc9d365004bc1a4d66"
server_key_iv: "fcb2097deabb3ebf8feeda3232593ad5"
server_key_version: "encryptionkey@0"
I20260928 08:57:42.630240 12631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "ac0f6b6a37864ff88931ab601ca0abb2"
format_stamp: "Formatted at 2026-09-28 08:57:42 on dist-test-slave-pbjp"
server_key: "95eb9dcbc10dbdfc9d365004bc1a4d66"
server_key_iv: "fcb2097deabb3ebf8feeda3232593ad5"
server_key_version: "encryptionkey@0"
I20260928 08:57:42.631513 12631 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 08:57:42.632196 12646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:42.632398 12631 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:57:42.632481 12631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "ac0f6b6a37864ff88931ab601ca0abb2"
format_stamp: "Formatted at 2026-09-28 08:57:42 on dist-test-slave-pbjp"
server_key: "95eb9dcbc10dbdfc9d365004bc1a4d66"
server_key_iv: "fcb2097deabb3ebf8feeda3232593ad5"
server_key_version: "encryptionkey@0"
I20260928 08:57:42.632550 12631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:42.640710 12631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:42.642409 12631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:42.642534 12631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:42.646212 12631 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39079
I20260928 08:57:42.646256 12698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39079 every 8 connection(s)
I20260928 08:57:42.646603 12631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:57:42.647496 12699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:42.649838 12699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac0f6b6a37864ff88931ab601ca0abb2: Bootstrap starting.
I20260928 08:57:42.650398 12699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac0f6b6a37864ff88931ab601ca0abb2: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:42.650648 12699 log.cc:824] T 00000000000000000000000000000000 P ac0f6b6a37864ff88931ab601ca0abb2: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:42.651314 12699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac0f6b6a37864ff88931ab601ca0abb2: No bootstrap required, opened a new log
I20260928 08:57:42.652916 12699 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac0f6b6a37864ff88931ab601ca0abb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac0f6b6a37864ff88931ab601ca0abb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39079 } }
I20260928 08:57:42.653091 12699 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac0f6b6a37864ff88931ab601ca0abb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:42.653128 12699 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac0f6b6a37864ff88931ab601ca0abb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac0f6b6a37864ff88931ab601ca0abb2, State: Initialized, Role: FOLLOWER
I20260928 08:57:42.653262 12699 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac0f6b6a37864ff88931ab601ca0abb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac0f6b6a37864ff88931ab601ca0abb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39079 } }
I20260928 08:57:42.653327 12699 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac0f6b6a37864ff88931ab601ca0abb2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:57:42.653350 12699 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac0f6b6a37864ff88931ab601ca0abb2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:57:42.653400 12699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac0f6b6a37864ff88931ab601ca0abb2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:42.653926 12699 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac0f6b6a37864ff88931ab601ca0abb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac0f6b6a37864ff88931ab601ca0abb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39079 } }
I20260928 08:57:42.654016 12699 leader_election.cc:302] T 00000000000000000000000000000000 P ac0f6b6a37864ff88931ab601ca0abb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac0f6b6a37864ff88931ab601ca0abb2; no voters:
I20260928 08:57:42.654170 12699 leader_election.cc:288] T 00000000000000000000000000000000 P ac0f6b6a37864ff88931ab601ca0abb2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:57:42.654222 12702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac0f6b6a37864ff88931ab601ca0abb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:57:42.654402 12699 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac0f6b6a37864ff88931ab601ca0abb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:57:42.654443 12702 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac0f6b6a37864ff88931ab601ca0abb2 [term 1 LEADER]: Becoming Leader. State: Replica: ac0f6b6a37864ff88931ab601ca0abb2, State: Running, Role: LEADER
I20260928 08:57:42.654584 12702 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac0f6b6a37864ff88931ab601ca0abb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac0f6b6a37864ff88931ab601ca0abb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39079 } }
I20260928 08:57:42.655179 12704 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac0f6b6a37864ff88931ab601ca0abb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac0f6b6a37864ff88931ab601ca0abb2. Latest consensus state: current_term: 1 leader_uuid: "ac0f6b6a37864ff88931ab601ca0abb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac0f6b6a37864ff88931ab601ca0abb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39079 } } }
I20260928 08:57:42.655194 12703 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac0f6b6a37864ff88931ab601ca0abb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac0f6b6a37864ff88931ab601ca0abb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac0f6b6a37864ff88931ab601ca0abb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39079 } } }
I20260928 08:57:42.655267 12704 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac0f6b6a37864ff88931ab601ca0abb2 [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:42.655283 12703 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac0f6b6a37864ff88931ab601ca0abb2 [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:42.655864 12712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:57:42.656380 12712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:57:42.656666 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 12631
I20260928 08:57:42.656795 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:57:42.656976 24223 external_mini_cluster.cc:1428] Setting key bfc1b7e1eb2797d6b71c7a2e9630674c
I20260928 08:57:42.658138 12712 catalog_manager.cc:1347] Generated new cluster ID: 02954225a3524f4e9d77e205bca3a309
I20260928 08:57:42.658192 12712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 08:57:42 dist-test-slave-pbjp krb5kdc[12618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 08:57:42.668889 12721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:42.657498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:43773 (local address 127.0.0.1:39079)
0928 08:57:42.657679 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:42.657689 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:42.657873 (+ 184us) server_negotiation.cc:407] Connection header received
0928 08:57:42.658000 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:42.658006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:42.658055 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:42.658182 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:42.659067 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:42.660540 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:42.661602 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:42.662252 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:42.662370 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:42.665173 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:42.665177 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:42.665178 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:42.665191 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:42.667045 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:42.667494 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:42.667497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:42.667498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:42.667606 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:42.667825 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:42.667825 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:42.667825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:42.668119 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:42.668263 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:42.668404 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 08:57:42.668620 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":136,"thread_start_us":79,"threads_started":1}
I20260928 08:57:42.669461 12712 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:57:42.670074 12712 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:57:42.670153 24223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:57:42.679275 12712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac0f6b6a37864ff88931ab601ca0abb2: Generated new TSK 0
I20260928 08:57:42.679479 12712 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:57:42.680353 12721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:42.671542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:55167 (local address 127.0.0.1:39079)
0928 08:57:42.671575 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:42.671576 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:42.673198 (+ 1622us) server_negotiation.cc:407] Connection header received
0928 08:57:42.673288 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:42.673289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:42.673300 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:42.673360 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:42.673940 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:42.674950 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:42.675930 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:42.676271 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:42.676338 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:42.677647 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:42.677649 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:42.677649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:42.677656 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:42.679138 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:42.679626 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:42.679627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:42.679627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:42.679697 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:42.679894 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:42.679895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:42.679895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:42.680022 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:42.680130 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:42.680204 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 08:57:42.680264 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 08:57:42.682097 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 12631
W20260928 08:57:42.685884 12616 connection.cc:535] client connection to 127.0.0.1:39079 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39079): Connection reset by peer (error 104)
2026-09-28T08:57:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1214 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:57:42 dist-test-slave-pbjp krb5kdc[12730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:57:42 dist-test-slave-pbjp krb5kdc[12730](info): set up 2 sockets
Sep 28 08:57:42 dist-test-slave-pbjp krb5kdc[12730](info): commencing operation
krb5kdc: starting...
W20260928 08:57:43.629281 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:57:43 dist-test-slave-pbjp krb5kdc[12730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:57:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:57:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:57:43.743482 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42969
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:43.823755 12743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:43.823963 12743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:43.823984 12743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:43.825145 12743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:57:43.825188 12743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:43.825201 12743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:43.825212 12743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:57:43.825222 12743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:57:43.826689 12743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:43.826916 12743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:43.827127 12743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:43.831738 12748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:43.831779 12743 server_base.cc:1034] running on GCE node
W20260928 08:57:43.831743 12749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:43.831743 12751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:43.832199 12743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:43.832475 12743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:43.833621 12743 hybrid_clock.cc:648] HybridClock initialized: now 1790585863833602 us; error 27 us; skew 500 ppm
Sep 28 08:57:43 dist-test-slave-pbjp krb5kdc[12730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:43.836006 12743 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:57:43.836395 12743 webserver.cc:460] Webserver started at http://127.0.0.1:37425/ using document root <none> and password file <none>
I20260928 08:57:43.836583 12743 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:43.836621 12743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:43.836727 12743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:43.837752 12743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "0d8c1bfe40144e57997e597c8d486560"
format_stamp: "Formatted at 2026-09-28 08:57:43 on dist-test-slave-pbjp"
server_key: "85cac2de211be875d879b2e1692ddf69"
server_key_iv: "03ece2eb4211c8c8be481b4bcea7ea9c"
server_key_version: "encryptionkey@0"
I20260928 08:57:43.838047 12743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "0d8c1bfe40144e57997e597c8d486560"
format_stamp: "Formatted at 2026-09-28 08:57:43 on dist-test-slave-pbjp"
server_key: "85cac2de211be875d879b2e1692ddf69"
server_key_iv: "03ece2eb4211c8c8be481b4bcea7ea9c"
server_key_version: "encryptionkey@0"
I20260928 08:57:43.839207 12743 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:57:43.839856 12758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:43.840009 12743 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:57:43.840087 12743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "0d8c1bfe40144e57997e597c8d486560"
format_stamp: "Formatted at 2026-09-28 08:57:43 on dist-test-slave-pbjp"
server_key: "85cac2de211be875d879b2e1692ddf69"
server_key_iv: "03ece2eb4211c8c8be481b4bcea7ea9c"
server_key_version: "encryptionkey@0"
I20260928 08:57:43.840145 12743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:43.852106 12743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:43.854471 12743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:43.854625 12743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:43.858484 12743 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42969
I20260928 08:57:43.858541 12810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42969 every 8 connection(s)
I20260928 08:57:43.858830 12743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:57:43.859700 12811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:43.862347 12811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d8c1bfe40144e57997e597c8d486560: Bootstrap starting.
I20260928 08:57:43.862948 12811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d8c1bfe40144e57997e597c8d486560: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:43.863214 12811 log.cc:824] T 00000000000000000000000000000000 P 0d8c1bfe40144e57997e597c8d486560: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:43.864002 12811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d8c1bfe40144e57997e597c8d486560: No bootstrap required, opened a new log
I20260928 08:57:43.865705 12811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d8c1bfe40144e57997e597c8d486560 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8c1bfe40144e57997e597c8d486560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42969 } }
I20260928 08:57:43.865856 12811 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d8c1bfe40144e57997e597c8d486560 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:43.865901 12811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d8c1bfe40144e57997e597c8d486560 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d8c1bfe40144e57997e597c8d486560, State: Initialized, Role: FOLLOWER
I20260928 08:57:43.866057 12811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d8c1bfe40144e57997e597c8d486560 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8c1bfe40144e57997e597c8d486560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42969 } }
I20260928 08:57:43.866139 12811 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d8c1bfe40144e57997e597c8d486560 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:57:43.866173 12811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d8c1bfe40144e57997e597c8d486560 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:57:43.866225 12811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d8c1bfe40144e57997e597c8d486560 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:43.866789 12811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d8c1bfe40144e57997e597c8d486560 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8c1bfe40144e57997e597c8d486560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42969 } }
I20260928 08:57:43.866899 12811 leader_election.cc:302] T 00000000000000000000000000000000 P 0d8c1bfe40144e57997e597c8d486560 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d8c1bfe40144e57997e597c8d486560; no voters:
I20260928 08:57:43.867069 12811 leader_election.cc:288] T 00000000000000000000000000000000 P 0d8c1bfe40144e57997e597c8d486560 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:57:43.867142 12814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d8c1bfe40144e57997e597c8d486560 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:57:43.867308 12814 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d8c1bfe40144e57997e597c8d486560 [term 1 LEADER]: Becoming Leader. State: Replica: 0d8c1bfe40144e57997e597c8d486560, State: Running, Role: LEADER
I20260928 08:57:43.867329 12811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d8c1bfe40144e57997e597c8d486560 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:57:43.867471 12814 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d8c1bfe40144e57997e597c8d486560 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8c1bfe40144e57997e597c8d486560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42969 } }
I20260928 08:57:43.867851 12815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d8c1bfe40144e57997e597c8d486560 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d8c1bfe40144e57997e597c8d486560" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8c1bfe40144e57997e597c8d486560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42969 } } }
I20260928 08:57:43.867869 12816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d8c1bfe40144e57997e597c8d486560 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d8c1bfe40144e57997e597c8d486560. Latest consensus state: current_term: 1 leader_uuid: "0d8c1bfe40144e57997e597c8d486560" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8c1bfe40144e57997e597c8d486560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42969 } } }
I20260928 08:57:43.867924 12815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d8c1bfe40144e57997e597c8d486560 [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:43.867949 12816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d8c1bfe40144e57997e597c8d486560 [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:43.868144 12818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:57:43.868304 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 12743
I20260928 08:57:43.868420 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:57:43.868638 24223 external_mini_cluster.cc:1428] Setting key afe0e8f40b31c25ff25398cb4307f543
I20260928 08:57:43.868834 12818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:57:43.870486 12818 catalog_manager.cc:1347] Generated new cluster ID: 113aaadd6a7d43f0b5319a93e17f7a54
I20260928 08:57:43.870535 12818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:57:43.875083 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:43.869151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:47669 (local address 127.0.0.1:42969)
0928 08:57:43.869719 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:43.869728 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:43.869752 (+ 24us) server_negotiation.cc:407] Connection header received
0928 08:57:43.869858 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:43.869876 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:43.869942 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:43.870080 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:43.870275 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:43.871707 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:43.872873 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:43.873530 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:43.873630 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:43.874390 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:43.874393 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:43.874394 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:43.874604 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:43.874643 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:43.874658 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:43.874732 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:43.874839 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 08:57:43.874874 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":93,"threads_started":1}
I20260928 08:57:43.875989 24223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:57:43.878252 12818 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:57:43.878998 12818 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:57:43.881189 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:43.877531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:51313 (local address 127.0.0.1:42969)
0928 08:57:43.877604 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:43.877606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:43.877657 (+ 51us) server_negotiation.cc:407] Connection header received
0928 08:57:43.877719 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:43.877720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:43.877739 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:43.877791 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:43.878153 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:43.878883 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:43.879948 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:43.880238 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:43.880293 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:43.880932 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:43.880933 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:43.880933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:43.880938 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:43.880954 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:43.880958 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:43.881009 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:43.881106 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 08:57:43.881118 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 08:57:43.883517 12818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d8c1bfe40144e57997e597c8d486560: Generated new TSK 0
I20260928 08:57:43.883662 12818 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:57:43.889374 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 12743
W20260928 08:57:43.893234 12728 connection.cc:535] client connection to 127.0.0.1:42969 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42969): Connection reset by peer (error 104)
2026-09-28T08:57:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1207 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:57:43 dist-test-slave-pbjp krb5kdc[12842](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:57:43 dist-test-slave-pbjp krb5kdc[12842](info): set up 2 sockets
Sep 28 08:57:43 dist-test-slave-pbjp krb5kdc[12842](info): commencing operation
krb5kdc: starting...
W20260928 08:57:44.841008 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:57:44 dist-test-slave-pbjp krb5kdc[12842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:57:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:57:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:57:44.954092 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34641
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:45.034751 12855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:45.035020 12855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:45.035048 12855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:45.036249 12855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:57:45.036294 12855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:45.036307 12855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:45.036319 12855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:57:45.036332 12855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:57:45.037873 12855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:45.038127 12855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:45.038317 12855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:45.042995 12860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:45.043025 12863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:45.043032 12861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:45.043180 12855 server_base.cc:1034] running on GCE node
I20260928 08:57:45.043486 12855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:45.043807 12855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:45.044955 12855 hybrid_clock.cc:648] HybridClock initialized: now 1790585865044939 us; error 33 us; skew 500 ppm
Sep 28 08:57:45 dist-test-slave-pbjp krb5kdc[12842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:45.047429 12855 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:57:45.047856 12855 webserver.cc:460] Webserver started at http://127.0.0.1:34477/ using document root <none> and password file <none>
I20260928 08:57:45.048095 12855 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:45.048146 12855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:45.048271 12855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:45.049376 12855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "105701a37a5b4a44ad5b3620e33d6f3c"
format_stamp: "Formatted at 2026-09-28 08:57:45 on dist-test-slave-pbjp"
server_key: "6b1224bbbedac0ac9c2f65664c2b357f"
server_key_iv: "d9203d5820ab6a16b857f0e4b86b69fd"
server_key_version: "encryptionkey@0"
I20260928 08:57:45.049710 12855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "105701a37a5b4a44ad5b3620e33d6f3c"
format_stamp: "Formatted at 2026-09-28 08:57:45 on dist-test-slave-pbjp"
server_key: "6b1224bbbedac0ac9c2f65664c2b357f"
server_key_iv: "d9203d5820ab6a16b857f0e4b86b69fd"
server_key_version: "encryptionkey@0"
I20260928 08:57:45.051054 12855 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:57:45.051781 12870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:45.051937 12855 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:57:45.052019 12855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "105701a37a5b4a44ad5b3620e33d6f3c"
format_stamp: "Formatted at 2026-09-28 08:57:45 on dist-test-slave-pbjp"
server_key: "6b1224bbbedac0ac9c2f65664c2b357f"
server_key_iv: "d9203d5820ab6a16b857f0e4b86b69fd"
server_key_version: "encryptionkey@0"
I20260928 08:57:45.052079 12855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:45.055336 12855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:45.057055 12855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:45.057204 12855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:45.060817 12855 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34641
I20260928 08:57:45.060848 12922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34641 every 8 connection(s)
I20260928 08:57:45.061210 12855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:57:45.062217 12923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:45.064467 12923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 105701a37a5b4a44ad5b3620e33d6f3c: Bootstrap starting.
I20260928 08:57:45.065023 12923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 105701a37a5b4a44ad5b3620e33d6f3c: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:45.065328 12923 log.cc:824] T 00000000000000000000000000000000 P 105701a37a5b4a44ad5b3620e33d6f3c: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:45.066020 12923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 105701a37a5b4a44ad5b3620e33d6f3c: No bootstrap required, opened a new log
I20260928 08:57:45.067622 12923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 105701a37a5b4a44ad5b3620e33d6f3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105701a37a5b4a44ad5b3620e33d6f3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34641 } }
I20260928 08:57:45.067760 12923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 105701a37a5b4a44ad5b3620e33d6f3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:45.067782 12923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 105701a37a5b4a44ad5b3620e33d6f3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 105701a37a5b4a44ad5b3620e33d6f3c, State: Initialized, Role: FOLLOWER
I20260928 08:57:45.067911 12923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 105701a37a5b4a44ad5b3620e33d6f3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105701a37a5b4a44ad5b3620e33d6f3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34641 } }
I20260928 08:57:45.067974 12923 raft_consensus.cc:388] T 00000000000000000000000000000000 P 105701a37a5b4a44ad5b3620e33d6f3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:57:45.067994 12923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 105701a37a5b4a44ad5b3620e33d6f3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:57:45.068040 12923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 105701a37a5b4a44ad5b3620e33d6f3c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:45.068343 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 12855
I20260928 08:57:45.068445 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:57:45.068629 24223 external_mini_cluster.cc:1428] Setting key 41380e9194f0ea86b6054f4c66011f55
I20260928 08:57:45.068638 12923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 105701a37a5b4a44ad5b3620e33d6f3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105701a37a5b4a44ad5b3620e33d6f3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34641 } }
I20260928 08:57:45.068738 12923 leader_election.cc:302] T 00000000000000000000000000000000 P 105701a37a5b4a44ad5b3620e33d6f3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 105701a37a5b4a44ad5b3620e33d6f3c; no voters:
I20260928 08:57:45.068894 12923 leader_election.cc:288] T 00000000000000000000000000000000 P 105701a37a5b4a44ad5b3620e33d6f3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:57:45.068920 12926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 105701a37a5b4a44ad5b3620e33d6f3c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:57:45.069129 12926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 105701a37a5b4a44ad5b3620e33d6f3c [term 1 LEADER]: Becoming Leader. State: Replica: 105701a37a5b4a44ad5b3620e33d6f3c, State: Running, Role: LEADER
I20260928 08:57:45.069233 12923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 105701a37a5b4a44ad5b3620e33d6f3c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:57:45.069252 12926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 105701a37a5b4a44ad5b3620e33d6f3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105701a37a5b4a44ad5b3620e33d6f3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34641 } }
I20260928 08:57:45.069664 12927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105701a37a5b4a44ad5b3620e33d6f3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 105701a37a5b4a44ad5b3620e33d6f3c. Latest consensus state: current_term: 1 leader_uuid: "105701a37a5b4a44ad5b3620e33d6f3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105701a37a5b4a44ad5b3620e33d6f3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34641 } } }
I20260928 08:57:45.069741 12927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105701a37a5b4a44ad5b3620e33d6f3c [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:45.070514 12932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:57:45.069931 12929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105701a37a5b4a44ad5b3620e33d6f3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "105701a37a5b4a44ad5b3620e33d6f3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105701a37a5b4a44ad5b3620e33d6f3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34641 } } }
I20260928 08:57:45.070660 12929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105701a37a5b4a44ad5b3620e33d6f3c [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:45.071092 12930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:45.069139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:54487 (local address 127.0.0.1:34641)
0928 08:57:45.069509 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:45.069518 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:45.069550 (+ 32us) server_negotiation.cc:407] Connection header received
0928 08:57:45.069673 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:45.069680 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:45.069711 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:45.070143 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:45.070569 (+ 426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:45.070573 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:45.070574 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:45.070578 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:45.070620 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:45.070635 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:45.070831 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:45.070865 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 08:57:45.070910 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":75,"threads_started":1}
I20260928 08:57:45.071220 12932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:57:45.071941 24223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:57:45.072721 12932 catalog_manager.cc:1347] Generated new cluster ID: e06f52f6ac1845efa84527b47d8d6b1b
I20260928 08:57:45.072772 12932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:57:45.078307 12930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:45.073366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:43433 (local address 127.0.0.1:34641)
0928 08:57:45.073406 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:45.073407 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:45.077376 (+ 3969us) server_negotiation.cc:407] Connection header received
0928 08:57:45.077462 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:45.077463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:45.077470 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:45.077499 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:45.078092 (+ 593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:45.078093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:45.078094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:45.078098 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:45.078114 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:45.078119 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:45.078167 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:45.078217 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 08:57:45.078232 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 08:57:45.083199 12932 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:57:45.083385 12932 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:57:45.092657 12932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 105701a37a5b4a44ad5b3620e33d6f3c: Generated new TSK 0
I20260928 08:57:45.092872 12932 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:57:45.100301 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 12855
2026-09-28T08:57:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1210 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:57:45 dist-test-slave-pbjp krb5kdc[12954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:57:45 dist-test-slave-pbjp krb5kdc[12954](info): set up 2 sockets
Sep 28 08:57:45 dist-test-slave-pbjp krb5kdc[12954](info): commencing operation
krb5kdc: starting...
W20260928 08:57:46.044907 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:57:46 dist-test-slave-pbjp krb5kdc[12954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:57:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:57:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:57:46.160317 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43131
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:46.243656 12967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:46.243875 12967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:46.243906 12967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:46.245146 12967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:57:46.245189 12967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:46.245208 12967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:46.245224 12967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:57:46.245239 12967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:57:46.246801 12967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43131
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:46.247063 12967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:46.247270 12967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:46.251873 12973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:46.251881 12972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:46.251968 12967 server_base.cc:1034] running on GCE node
W20260928 08:57:46.251883 12975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:46.252324 12967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:46.252606 12967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:46.253782 12967 hybrid_clock.cc:648] HybridClock initialized: now 1790585866253764 us; error 32 us; skew 500 ppm
Sep 28 08:57:46 dist-test-slave-pbjp krb5kdc[12954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:46.256224 12967 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:57:46.256664 12967 webserver.cc:460] Webserver started at http://127.0.0.1:40679/ using document root <none> and password file <none>
I20260928 08:57:46.256860 12967 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:46.256904 12967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:46.257036 12967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:46.258098 12967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "69ed76cf495d4f15badac1b648a54a54"
format_stamp: "Formatted at 2026-09-28 08:57:46 on dist-test-slave-pbjp"
server_key: "6e4d1b1e1a41ee17f31b46bbd25aed4e"
server_key_iv: "b8a377f25a92829412c6498f00a7274b"
server_key_version: "encryptionkey@0"
I20260928 08:57:46.258431 12967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "69ed76cf495d4f15badac1b648a54a54"
format_stamp: "Formatted at 2026-09-28 08:57:46 on dist-test-slave-pbjp"
server_key: "6e4d1b1e1a41ee17f31b46bbd25aed4e"
server_key_iv: "b8a377f25a92829412c6498f00a7274b"
server_key_version: "encryptionkey@0"
I20260928 08:57:46.259670 12967 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:57:46.260421 12982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:46.260564 12967 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:57:46.260644 12967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "69ed76cf495d4f15badac1b648a54a54"
format_stamp: "Formatted at 2026-09-28 08:57:46 on dist-test-slave-pbjp"
server_key: "6e4d1b1e1a41ee17f31b46bbd25aed4e"
server_key_iv: "b8a377f25a92829412c6498f00a7274b"
server_key_version: "encryptionkey@0"
I20260928 08:57:46.260704 12967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:46.270731 12967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:46.272461 12967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:46.272596 12967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:46.276466 12967 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43131
I20260928 08:57:46.276518 13034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43131 every 8 connection(s)
I20260928 08:57:46.276847 12967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:57:46.277725 13035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:46.280050 13035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69ed76cf495d4f15badac1b648a54a54: Bootstrap starting.
I20260928 08:57:46.280582 13035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69ed76cf495d4f15badac1b648a54a54: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:46.280813 13035 log.cc:824] T 00000000000000000000000000000000 P 69ed76cf495d4f15badac1b648a54a54: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:46.281471 13035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69ed76cf495d4f15badac1b648a54a54: No bootstrap required, opened a new log
I20260928 08:57:46.283033 13035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69ed76cf495d4f15badac1b648a54a54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ed76cf495d4f15badac1b648a54a54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43131 } }
I20260928 08:57:46.283176 13035 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69ed76cf495d4f15badac1b648a54a54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:46.283196 13035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69ed76cf495d4f15badac1b648a54a54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ed76cf495d4f15badac1b648a54a54, State: Initialized, Role: FOLLOWER
I20260928 08:57:46.283310 13035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69ed76cf495d4f15badac1b648a54a54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ed76cf495d4f15badac1b648a54a54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43131 } }
I20260928 08:57:46.283385 13035 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69ed76cf495d4f15badac1b648a54a54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:57:46.283413 13035 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69ed76cf495d4f15badac1b648a54a54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:57:46.283445 13035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69ed76cf495d4f15badac1b648a54a54 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:46.283936 13035 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69ed76cf495d4f15badac1b648a54a54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ed76cf495d4f15badac1b648a54a54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43131 } }
I20260928 08:57:46.284034 13035 leader_election.cc:302] T 00000000000000000000000000000000 P 69ed76cf495d4f15badac1b648a54a54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69ed76cf495d4f15badac1b648a54a54; no voters:
I20260928 08:57:46.284154 13035 leader_election.cc:288] T 00000000000000000000000000000000 P 69ed76cf495d4f15badac1b648a54a54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:57:46.284214 13038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69ed76cf495d4f15badac1b648a54a54 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:57:46.284379 13035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69ed76cf495d4f15badac1b648a54a54 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:57:46.284435 13038 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69ed76cf495d4f15badac1b648a54a54 [term 1 LEADER]: Becoming Leader. State: Replica: 69ed76cf495d4f15badac1b648a54a54, State: Running, Role: LEADER
I20260928 08:57:46.284533 13038 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69ed76cf495d4f15badac1b648a54a54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ed76cf495d4f15badac1b648a54a54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43131 } }
I20260928 08:57:46.284893 13039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69ed76cf495d4f15badac1b648a54a54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69ed76cf495d4f15badac1b648a54a54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ed76cf495d4f15badac1b648a54a54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43131 } } }
I20260928 08:57:46.285219 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 12967
I20260928 08:57:46.285305 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:57:46.285494 24223 external_mini_cluster.cc:1428] Setting key 44673134306bc43dd9316c91f870c764
I20260928 08:57:46.286106 13039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69ed76cf495d4f15badac1b648a54a54 [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:46.285049 13040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69ed76cf495d4f15badac1b648a54a54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69ed76cf495d4f15badac1b648a54a54. Latest consensus state: current_term: 1 leader_uuid: "69ed76cf495d4f15badac1b648a54a54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ed76cf495d4f15badac1b648a54a54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43131 } } }
W20260928 08:57:46.286339 13053 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 69ed76cf495d4f15badac1b648a54a54: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 08:57:46.286386 13040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69ed76cf495d4f15badac1b648a54a54 [sys.catalog]: This master's current role is: LEADER
W20260928 08:57:46.286389 13053 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:57:46.287587 13057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:57:46.287786 13057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:57:46.289350 13057 catalog_manager.cc:1347] Generated new cluster ID: c0f1f7ef5f40433cb7a66c4056f58abb
I20260928 08:57:46.289399 13057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 08:57:46 dist-test-slave-pbjp krb5kdc[12954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 08:57:46.296770 13055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:46.285972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:55201 (local address 127.0.0.1:43131)
0928 08:57:46.286402 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:46.286413 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:46.286450 (+ 37us) server_negotiation.cc:407] Connection header received
0928 08:57:46.286574 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:46.286580 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:46.286634 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:46.286749 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:46.287501 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:46.288558 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:46.289657 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:46.290250 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:46.290348 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:46.293138 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:46.293144 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:46.293145 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:46.293160 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:46.295067 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:46.295495 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:46.295500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:46.295500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:46.295643 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:46.295847 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:46.295848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:46.295848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:46.296177 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:46.296345 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:46.296425 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 08:57:46.296531 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":90,"threads_started":1}
I20260928 08:57:46.297767 24223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:57:46.298897 13057 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:57:46.299639 13057 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:57:46.306469 13055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:46.299223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:57473 (local address 127.0.0.1:43131)
0928 08:57:46.299259 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:46.299261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:46.299397 (+ 136us) server_negotiation.cc:407] Connection header received
0928 08:57:46.299443 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:46.299444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:46.299654 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:46.299710 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:46.300390 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:46.301175 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:46.302207 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:46.302497 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:46.302549 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:46.303846 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:46.303847 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:46.303847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:46.303854 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:46.305305 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:46.305687 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:46.305688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:46.305688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:46.305735 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:46.305934 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:46.305935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:46.305935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:46.306042 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:46.306191 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:46.306323 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 08:57:46.306379 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 08:57:46.312312 13057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69ed76cf495d4f15badac1b648a54a54: Generated new TSK 0
I20260928 08:57:46.312528 13057 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:57:46.318679 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 12967
W20260928 08:57:46.322542 12952 connection.cc:535] client connection to 127.0.0.1:43131 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43131): Connection reset by peer (error 104)
2026-09-28T08:57:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1218 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:57:46 dist-test-slave-pbjp krb5kdc[13066](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:57:46 dist-test-slave-pbjp krb5kdc[13066](info): set up 2 sockets
Sep 28 08:57:46 dist-test-slave-pbjp krb5kdc[13066](info): commencing operation
krb5kdc: starting...
W20260928 08:57:47.260988 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:57:47 dist-test-slave-pbjp krb5kdc[13066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:57:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:57:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:57:47.379114 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45531
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:47.459681 13079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:47.459915 13079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:47.459940 13079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:47.461148 13079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:57:47.461189 13079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:47.461201 13079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:47.461213 13079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:57:47.461223 13079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:57:47.462736 13079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45531
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:47.462949 13079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:47.463145 13079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:47.468176 13087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:47.468288 13084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:47.468171 13085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:47.468451 13079 server_base.cc:1034] running on GCE node
I20260928 08:57:47.468703 13079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:47.468999 13079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:47.470149 13079 hybrid_clock.cc:648] HybridClock initialized: now 1790585867470141 us; error 42 us; skew 500 ppm
Sep 28 08:57:47 dist-test-slave-pbjp krb5kdc[13066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:47.472584 13079 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:57:47.473039 13079 webserver.cc:460] Webserver started at http://127.0.0.1:42297/ using document root <none> and password file <none>
I20260928 08:57:47.473269 13079 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:47.473312 13079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:47.473425 13079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:47.474450 13079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "f46cf08180004725bef2bb35f368fdb9"
format_stamp: "Formatted at 2026-09-28 08:57:47 on dist-test-slave-pbjp"
server_key: "24a7ef5724cdc05eeaac2984c011e60d"
server_key_iv: "846c466f8d6a2c18fcf6c43dce3cd39b"
server_key_version: "encryptionkey@0"
I20260928 08:57:47.474764 13079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "f46cf08180004725bef2bb35f368fdb9"
format_stamp: "Formatted at 2026-09-28 08:57:47 on dist-test-slave-pbjp"
server_key: "24a7ef5724cdc05eeaac2984c011e60d"
server_key_iv: "846c466f8d6a2c18fcf6c43dce3cd39b"
server_key_version: "encryptionkey@0"
I20260928 08:57:47.475997 13079 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:57:47.476625 13094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:47.476784 13079 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:57:47.476863 13079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "f46cf08180004725bef2bb35f368fdb9"
format_stamp: "Formatted at 2026-09-28 08:57:47 on dist-test-slave-pbjp"
server_key: "24a7ef5724cdc05eeaac2984c011e60d"
server_key_iv: "846c466f8d6a2c18fcf6c43dce3cd39b"
server_key_version: "encryptionkey@0"
I20260928 08:57:47.476917 13079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:47.502864 13079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:47.504923 13079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:47.505048 13079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:47.508737 13079 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45531
I20260928 08:57:47.508795 13146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45531 every 8 connection(s)
I20260928 08:57:47.509143 13079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:57:47.510020 13147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:47.512285 13147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f46cf08180004725bef2bb35f368fdb9: Bootstrap starting.
I20260928 08:57:47.512837 13147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f46cf08180004725bef2bb35f368fdb9: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:47.513139 13147 log.cc:824] T 00000000000000000000000000000000 P f46cf08180004725bef2bb35f368fdb9: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:47.513824 13147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f46cf08180004725bef2bb35f368fdb9: No bootstrap required, opened a new log
I20260928 08:57:47.514000 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 13079
I20260928 08:57:47.514107 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:57:47.514276 24223 external_mini_cluster.cc:1428] Setting key 0e8dc57d0ee7ea74c08603aeea3bcc27
I20260928 08:57:47.515937 13147 raft_consensus.cc:348] T 00000000000000000000000000000000 P f46cf08180004725bef2bb35f368fdb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46cf08180004725bef2bb35f368fdb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45531 } }
I20260928 08:57:47.516129 13147 raft_consensus.cc:374] T 00000000000000000000000000000000 P f46cf08180004725bef2bb35f368fdb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:47.516168 13147 raft_consensus.cc:729] T 00000000000000000000000000000000 P f46cf08180004725bef2bb35f368fdb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46cf08180004725bef2bb35f368fdb9, State: Initialized, Role: FOLLOWER
I20260928 08:57:47.516351 13147 consensus_queue.cc:260] T 00000000000000000000000000000000 P f46cf08180004725bef2bb35f368fdb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46cf08180004725bef2bb35f368fdb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45531 } }
I20260928 08:57:47.516440 13147 raft_consensus.cc:388] T 00000000000000000000000000000000 P f46cf08180004725bef2bb35f368fdb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:57:47.516471 13147 raft_consensus.cc:482] T 00000000000000000000000000000000 P f46cf08180004725bef2bb35f368fdb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:57:47.516517 13147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f46cf08180004725bef2bb35f368fdb9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:47.517302 13147 raft_consensus.cc:504] T 00000000000000000000000000000000 P f46cf08180004725bef2bb35f368fdb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46cf08180004725bef2bb35f368fdb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45531 } }
I20260928 08:57:47.517412 13147 leader_election.cc:302] T 00000000000000000000000000000000 P f46cf08180004725bef2bb35f368fdb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f46cf08180004725bef2bb35f368fdb9; no voters:
I20260928 08:57:47.517565 13147 leader_election.cc:288] T 00000000000000000000000000000000 P f46cf08180004725bef2bb35f368fdb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:57:47.517603 13152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f46cf08180004725bef2bb35f368fdb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:57:47.517755 13152 raft_consensus.cc:686] T 00000000000000000000000000000000 P f46cf08180004725bef2bb35f368fdb9 [term 1 LEADER]: Becoming Leader. State: Replica: f46cf08180004725bef2bb35f368fdb9, State: Running, Role: LEADER
I20260928 08:57:47.517797 13147 sys_catalog.cc:564] T 00000000000000000000000000000000 P f46cf08180004725bef2bb35f368fdb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:57:47.517907 13152 consensus_queue.cc:237] T 00000000000000000000000000000000 P f46cf08180004725bef2bb35f368fdb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46cf08180004725bef2bb35f368fdb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45531 } }
I20260928 08:57:47.518267 13153 sys_catalog.cc:455] T 00000000000000000000000000000000 P f46cf08180004725bef2bb35f368fdb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f46cf08180004725bef2bb35f368fdb9. Latest consensus state: current_term: 1 leader_uuid: "f46cf08180004725bef2bb35f368fdb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46cf08180004725bef2bb35f368fdb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45531 } } }
I20260928 08:57:47.518389 13153 sys_catalog.cc:458] T 00000000000000000000000000000000 P f46cf08180004725bef2bb35f368fdb9 [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:47.518589 13154 sys_catalog.cc:455] T 00000000000000000000000000000000 P f46cf08180004725bef2bb35f368fdb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f46cf08180004725bef2bb35f368fdb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46cf08180004725bef2bb35f368fdb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45531 } } }
I20260928 08:57:47.518685 13154 sys_catalog.cc:458] T 00000000000000000000000000000000 P f46cf08180004725bef2bb35f368fdb9 [sys.catalog]: This master's current role is: LEADER
W20260928 08:57:47.519574 13168 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f46cf08180004725bef2bb35f368fdb9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:57:47.520004 13168 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:57:47.520076 13160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:57:47.520243 13160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:57:47.521603 13160 catalog_manager.cc:1347] Generated new cluster ID: a4cccc6432374a65a576109013cc05df
I20260928 08:57:47.521646 13160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:57:47.526315 13151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:47.514786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:58025 (local address 127.0.0.1:45531)
0928 08:57:47.514981 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:47.514991 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:47.515029 (+ 38us) server_negotiation.cc:407] Connection header received
0928 08:57:47.515893 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:47.515903 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:47.515976 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:47.516137 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:47.516310 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:47.517984 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:47.519094 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:47.519553 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:47.519651 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:47.525620 (+ 5969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:47.525624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:47.525625 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:47.525840 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:47.525885 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:47.525902 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:47.525993 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:47.526076 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 08:57:47.526107 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":81,"threads_started":1}
I20260928 08:57:47.527475 24223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:57:47.532305 13151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:47.529059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:37093 (local address 127.0.0.1:45531)
0928 08:57:47.529116 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:47.529118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:47.529126 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:57:47.529144 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:47.529145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:47.529165 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:47.529218 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:47.529352 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:47.530102 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:47.531058 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:47.531380 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:47.531439 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:47.532068 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:47.532070 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:47.532070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:47.532075 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:47.532090 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:47.532095 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:47.532155 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:47.532215 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 08:57:47.532228 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260928 08:57:47.544939 13160 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:57:47.545687 13160 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:57:47.551059 13160 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f46cf08180004725bef2bb35f368fdb9: Generated new TSK 0
I20260928 08:57:47.551229 13160 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:57:47.554500 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 13079
W20260928 08:57:47.558578 13064 connection.cc:535] client connection to 127.0.0.1:45531 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45531): Connection reset by peer (error 104)
2026-09-28T08:57:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1235 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:57:47 dist-test-slave-pbjp krb5kdc[13178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:57:47 dist-test-slave-pbjp krb5kdc[13178](info): set up 2 sockets
Sep 28 08:57:47 dist-test-slave-pbjp krb5kdc[13178](info): commencing operation
krb5kdc: starting...
W20260928 08:57:48.501053 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:57:48 dist-test-slave-pbjp krb5kdc[13178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:57:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:57:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:57:48.612102 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42437
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:48.692648 13191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:48.692878 13191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:48.692910 13191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:48.694166 13191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:57:48.694211 13191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:48.694229 13191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:48.694244 13191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:57:48.694259 13191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:57:48.695770 13191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42437
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585589999956-24223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:48.696023 13191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:48.696238 13191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:48.701023 13199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:48.701188 13191 server_base.cc:1034] running on GCE node
W20260928 08:57:48.701179 13196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:48.701164 13197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:48.701542 13191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:48.701829 13191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:48.702970 13191 hybrid_clock.cc:648] HybridClock initialized: now 1790585868702948 us; error 32 us; skew 500 ppm
Sep 28 08:57:48 dist-test-slave-pbjp krb5kdc[13178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:48.710064 13191 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:57:48.710578 13191 webserver.cc:460] Webserver started at http://127.0.0.1:42853/ using document root <none> and password file <none>
I20260928 08:57:48.710799 13191 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:48.710851 13191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:48.710989 13191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:48.712159 13191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "61049bf1c1354d05ab80ba4973e1c4b0"
format_stamp: "Formatted at 2026-09-28 08:57:48 on dist-test-slave-pbjp"
server_key: "92dc4d5db34c1a1273dcfcb6bb494290"
server_key_iv: "5033dd7e4ee394e28d951ec7594cf3d5"
server_key_version: "encryptionkey@0"
I20260928 08:57:48.712596 13191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "61049bf1c1354d05ab80ba4973e1c4b0"
format_stamp: "Formatted at 2026-09-28 08:57:48 on dist-test-slave-pbjp"
server_key: "92dc4d5db34c1a1273dcfcb6bb494290"
server_key_iv: "5033dd7e4ee394e28d951ec7594cf3d5"
server_key_version: "encryptionkey@0"
I20260928 08:57:48.713958 13191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 08:57:48.714677 13206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:48.714828 13191 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:48.714915 13191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "61049bf1c1354d05ab80ba4973e1c4b0"
format_stamp: "Formatted at 2026-09-28 08:57:48 on dist-test-slave-pbjp"
server_key: "92dc4d5db34c1a1273dcfcb6bb494290"
server_key_iv: "5033dd7e4ee394e28d951ec7594cf3d5"
server_key_version: "encryptionkey@0"
I20260928 08:57:48.714977 13191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:48.718320 13191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:48.720252 13191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:48.720382 13191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:48.724018 13191 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42437
I20260928 08:57:48.724053 13258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42437 every 8 connection(s)
I20260928 08:57:48.724372 13191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:57:48.725378 13259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:48.726703 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 13191
I20260928 08:57:48.726804 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:57:48.726974 24223 external_mini_cluster.cc:1428] Setting key b8f667779966303859f6d69c916368ba
I20260928 08:57:48.727926 13259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61049bf1c1354d05ab80ba4973e1c4b0: Bootstrap starting.
I20260928 08:57:48.728621 13259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61049bf1c1354d05ab80ba4973e1c4b0: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:48.729252 13259 log.cc:824] T 00000000000000000000000000000000 P 61049bf1c1354d05ab80ba4973e1c4b0: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:48.729641 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:48.727525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:51723 (local address 127.0.0.1:42437)
0928 08:57:48.727714 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:48.727723 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:48.727756 (+ 33us) server_negotiation.cc:407] Connection header received
0928 08:57:48.728314 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:48.728325 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:48.728363 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:48.728518 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:48.729148 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:48.729152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:48.729152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:48.729157 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:48.729202 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:48.729216 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:48.729316 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:48.729447 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 08:57:48.729495 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260928 08:57:48.729967 13259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61049bf1c1354d05ab80ba4973e1c4b0: No bootstrap required, opened a new log
I20260928 08:57:48.730613 24223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:57:48.732074 13259 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61049bf1c1354d05ab80ba4973e1c4b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61049bf1c1354d05ab80ba4973e1c4b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42437 } }
I20260928 08:57:48.732257 13259 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61049bf1c1354d05ab80ba4973e1c4b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:48.732301 13259 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61049bf1c1354d05ab80ba4973e1c4b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61049bf1c1354d05ab80ba4973e1c4b0, State: Initialized, Role: FOLLOWER
I20260928 08:57:48.732468 13259 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61049bf1c1354d05ab80ba4973e1c4b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61049bf1c1354d05ab80ba4973e1c4b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42437 } }
I20260928 08:57:48.732568 13259 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61049bf1c1354d05ab80ba4973e1c4b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:57:48.732623 13259 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61049bf1c1354d05ab80ba4973e1c4b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:57:48.732677 13259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61049bf1c1354d05ab80ba4973e1c4b0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:48.732921 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:48.731911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:47833 (local address 127.0.0.1:42437)
0928 08:57:48.731943 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:48.731945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:48.732057 (+ 112us) server_negotiation.cc:407] Connection header received
0928 08:57:48.732116 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:48.732116 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:48.732120 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:48.732163 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:48.732731 (+ 568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:48.732732 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:48.732733 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:48.732737 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:48.732752 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:48.732756 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:48.732799 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:48.732868 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 08:57:48.732878 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 08:57:48.733421 13259 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61049bf1c1354d05ab80ba4973e1c4b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61049bf1c1354d05ab80ba4973e1c4b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42437 } }
I20260928 08:57:48.733521 13259 leader_election.cc:302] T 00000000000000000000000000000000 P 61049bf1c1354d05ab80ba4973e1c4b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61049bf1c1354d05ab80ba4973e1c4b0; no voters:
I20260928 08:57:48.733639 13259 leader_election.cc:288] T 00000000000000000000000000000000 P 61049bf1c1354d05ab80ba4973e1c4b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:57:48.733682 13269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61049bf1c1354d05ab80ba4973e1c4b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:57:48.733798 13269 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61049bf1c1354d05ab80ba4973e1c4b0 [term 1 LEADER]: Becoming Leader. State: Replica: 61049bf1c1354d05ab80ba4973e1c4b0, State: Running, Role: LEADER
I20260928 08:57:48.733897 13259 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61049bf1c1354d05ab80ba4973e1c4b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:57:48.733877 13269 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61049bf1c1354d05ab80ba4973e1c4b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61049bf1c1354d05ab80ba4973e1c4b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42437 } }
I20260928 08:57:48.734215 13269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61049bf1c1354d05ab80ba4973e1c4b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61049bf1c1354d05ab80ba4973e1c4b0. Latest consensus state: current_term: 1 leader_uuid: "61049bf1c1354d05ab80ba4973e1c4b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61049bf1c1354d05ab80ba4973e1c4b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42437 } } }
I20260928 08:57:48.734243 13271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61049bf1c1354d05ab80ba4973e1c4b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61049bf1c1354d05ab80ba4973e1c4b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61049bf1c1354d05ab80ba4973e1c4b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42437 } } }
I20260928 08:57:48.734377 13271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61049bf1c1354d05ab80ba4973e1c4b0 [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:48.734362 13269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61049bf1c1354d05ab80ba4973e1c4b0 [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:48.734627 13274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:57:48.735148 13274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:57:48.736577 13274 catalog_manager.cc:1347] Generated new cluster ID: 19ab630a0fd24e7593c20729877d9c6e
I20260928 08:57:48.736618 13274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:57:48.750715 13274 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:57:48.750909 13274 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:57:48.755280 13274 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61049bf1c1354d05ab80ba4973e1c4b0: Generated new TSK 0
I20260928 08:57:48.755528 13274 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:57:48.762231 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 13191
2026-09-28T08:57:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1207 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:57:48 dist-test-slave-pbjp krb5kdc[13290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:57:48 dist-test-slave-pbjp krb5kdc[13290](info): set up 2 sockets
Sep 28 08:57:48 dist-test-slave-pbjp krb5kdc[13290](info): commencing operation
krb5kdc: starting...
W20260928 08:57:49.708930 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:57:49 dist-test-slave-pbjp krb5kdc[13290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:57:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:57:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.167.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
I20260928 08:57:49.819705 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:43523
--webserver_interface=127.23.167.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:46145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:43523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:49.899436 13303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:49.899641 13303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:49.899664 13303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:49.900777 13303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:57:49.900818 13303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:49.900830 13303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:49.900841 13303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:57:49.900854 13303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:57:49.902307 13303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:46145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:43523
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:43523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:49.902524 13303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:49.902706 13303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:49.907486 13309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:49.907470 13308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:49.907470 13311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:49.907582 13303 server_base.cc:1034] running on GCE node
I20260928 08:57:49.907891 13303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:49.908139 13303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:49.909276 13303 hybrid_clock.cc:648] HybridClock initialized: now 1790585869909262 us; error 26 us; skew 500 ppm
Sep 28 08:57:49 dist-test-slave-pbjp krb5kdc[13290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585869, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:49.911779 13303 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:57:49.912227 13303 webserver.cc:460] Webserver started at http://127.23.167.254:42611/ using document root <none> and password file <none>
I20260928 08:57:49.912428 13303 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:49.912477 13303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:49.912601 13303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:49.913695 13303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "1c1f77756d324333a506350d447b69e1"
format_stamp: "Formatted at 2026-09-28 08:57:49 on dist-test-slave-pbjp"
server_key: "bb46abb8089e87754e9cce0f825bd3a1"
server_key_iv: "d9f17455da335ba854a5c8cd4c359ed2"
server_key_version: "encryptionkey@0"
I20260928 08:57:49.914018 13303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "1c1f77756d324333a506350d447b69e1"
format_stamp: "Formatted at 2026-09-28 08:57:49 on dist-test-slave-pbjp"
server_key: "bb46abb8089e87754e9cce0f825bd3a1"
server_key_iv: "d9f17455da335ba854a5c8cd4c359ed2"
server_key_version: "encryptionkey@0"
I20260928 08:57:49.915227 13303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:57:49.915915 13318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:49.916075 13303 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:57:49.916157 13303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "1c1f77756d324333a506350d447b69e1"
format_stamp: "Formatted at 2026-09-28 08:57:49 on dist-test-slave-pbjp"
server_key: "bb46abb8089e87754e9cce0f825bd3a1"
server_key_iv: "d9f17455da335ba854a5c8cd4c359ed2"
server_key_version: "encryptionkey@0"
I20260928 08:57:49.916214 13303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:49.926919 13303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:49.928995 13303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:49.929162 13303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:49.932848 13303 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:43523
I20260928 08:57:49.932881 13370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:43523 every 8 connection(s)
I20260928 08:57:49.933256 13303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:57:49.934096 13371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:49.934219 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 13303
I20260928 08:57:49.934325 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:57:49.934520 24223 external_mini_cluster.cc:1428] Setting key 916c819222b4ad5f64b6e425a871f98b
I20260928 08:57:49.937187 13371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c1f77756d324333a506350d447b69e1: Bootstrap starting.
I20260928 08:57:49.937824 13371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c1f77756d324333a506350d447b69e1: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:49.938114 13371 log.cc:824] T 00000000000000000000000000000000 P 1c1f77756d324333a506350d447b69e1: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:49.938795 13371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c1f77756d324333a506350d447b69e1: No bootstrap required, opened a new log
Sep 28 08:57:49 dist-test-slave-pbjp krb5kdc[13290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:57:49.940862 13371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c1f77756d324333a506350d447b69e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c1f77756d324333a506350d447b69e1" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43523 } }
I20260928 08:57:49.941027 13371 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c1f77756d324333a506350d447b69e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:49.941133 13371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c1f77756d324333a506350d447b69e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c1f77756d324333a506350d447b69e1, State: Initialized, Role: FOLLOWER
I20260928 08:57:49.941260 13371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c1f77756d324333a506350d447b69e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c1f77756d324333a506350d447b69e1" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43523 } }
I20260928 08:57:49.941324 13371 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c1f77756d324333a506350d447b69e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:57:49.941345 13371 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c1f77756d324333a506350d447b69e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:57:49.941397 13371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c1f77756d324333a506350d447b69e1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:49.941992 13371 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c1f77756d324333a506350d447b69e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c1f77756d324333a506350d447b69e1" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43523 } }
I20260928 08:57:49.942090 13371 leader_election.cc:302] T 00000000000000000000000000000000 P 1c1f77756d324333a506350d447b69e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c1f77756d324333a506350d447b69e1; no voters:
I20260928 08:57:49.942251 13371 leader_election.cc:288] T 00000000000000000000000000000000 P 1c1f77756d324333a506350d447b69e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:57:49.942399 13377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c1f77756d324333a506350d447b69e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:57:49.942528 13371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c1f77756d324333a506350d447b69e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:57:49.942552 13377 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c1f77756d324333a506350d447b69e1 [term 1 LEADER]: Becoming Leader. State: Replica: 1c1f77756d324333a506350d447b69e1, State: Running, Role: LEADER
I20260928 08:57:49.942682 13377 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c1f77756d324333a506350d447b69e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c1f77756d324333a506350d447b69e1" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43523 } }
I20260928 08:57:49.943017 13376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c1f77756d324333a506350d447b69e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c1f77756d324333a506350d447b69e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c1f77756d324333a506350d447b69e1" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43523 } } }
I20260928 08:57:49.943089 13376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c1f77756d324333a506350d447b69e1 [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:49.943244 13378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c1f77756d324333a506350d447b69e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c1f77756d324333a506350d447b69e1. Latest consensus state: current_term: 1 leader_uuid: "1c1f77756d324333a506350d447b69e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c1f77756d324333a506350d447b69e1" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43523 } } }
I20260928 08:57:49.943320 13378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c1f77756d324333a506350d447b69e1 [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:49.943826 13385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:57:49.944538 13385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:57:49.944880 13374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:49.935012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53197 (local address 127.23.167.254:43523)
0928 08:57:49.935192 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:49.935200 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:49.935226 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:57:49.935639 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:49.935646 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:49.935693 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:49.935806 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:49.936530 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:49.937492 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:49.938475 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:49.938900 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:49.938997 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:49.941494 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:49.941498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:49.941499 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:49.941510 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:49.943355 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:49.943775 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:49.943778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:49.943779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:49.943895 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:49.944133 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:49.944134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:49.944134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:49.944352 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:49.944511 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:49.944637 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 08:57:49.944729 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":76,"threads_started":1}
I20260928 08:57:49.945828 24223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:57:49.947132 13385 catalog_manager.cc:1347] Generated new cluster ID: 8ab0767198d84b0585bed93a0356555f
I20260928 08:57:49.947180 13385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:57:49.952405 13374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:49.947229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47491 (local address 127.23.167.254:43523)
0928 08:57:49.947265 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:49.947266 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:49.947324 (+ 58us) server_negotiation.cc:407] Connection header received
0928 08:57:49.947374 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:49.947375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:49.947384 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:49.947425 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:49.947877 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:49.948331 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:49.948967 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:49.949237 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:49.949290 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:49.950273 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:49.950275 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:49.950275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:49.950281 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:49.951430 (+ 1149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:49.951746 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:49.951747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:49.951747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:49.951796 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:49.951972 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:49.951973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:49.951973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:49.952072 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:49.952155 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:49.952270 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 08:57:49.952326 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 08:57:49.964473 13385 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:57:49.964944 13385 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:57:49.972474 13385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c1f77756d324333a506350d447b69e1: Generated new TSK 0
I20260928 08:57:49.972700 13385 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:57:49.974395 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 13303
W20260928 08:57:49.978315 13288 connection.cc:535] client connection to 127.23.167.254:43523 recv error: Network error: failed to read from TLS socket (remote: 127.23.167.254:43523): Connection reset by peer (error 104)
2026-09-28T08:57:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1212 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:57:50 dist-test-slave-pbjp krb5kdc[13402](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:57:50 dist-test-slave-pbjp krb5kdc[13402](info): set up 2 sockets
Sep 28 08:57:50 dist-test-slave-pbjp krb5kdc[13402](info): commencing operation
krb5kdc: starting...
W20260928 08:57:50.920928 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:57:50 dist-test-slave-pbjp krb5kdc[13402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:57:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:57:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.167.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
I20260928 08:57:51.035734 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:46581
--webserver_interface=127.23.167.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:46533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:46581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:51.114867 13415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:51.115116 13415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:51.115149 13415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:51.116369 13415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:57:51.116418 13415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:51.116438 13415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:51.116454 13415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:57:51.116469 13415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:57:51.118069 13415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:46533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:46581
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:46581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:51.118335 13415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:51.118546 13415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:51.123390 13420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:51.123540 13415 server_base.cc:1034] running on GCE node
W20260928 08:57:51.123464 13421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:51.123390 13423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:51.123931 13415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:51.124272 13415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:51.125479 13415 hybrid_clock.cc:648] HybridClock initialized: now 1790585871125456 us; error 40 us; skew 500 ppm
Sep 28 08:57:51 dist-test-slave-pbjp krb5kdc[13402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585871, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:51.127936 13415 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:57:51.128326 13415 webserver.cc:460] Webserver started at http://127.23.167.254:44195/ using document root <none> and password file <none>
I20260928 08:57:51.128556 13415 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:51.128597 13415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:51.128688 13415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:51.129765 13415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "10f6a30a72ba48e596737a30cfe91a71"
format_stamp: "Formatted at 2026-09-28 08:57:51 on dist-test-slave-pbjp"
server_key: "04928ff3d8056a9db9b4fbda92ae8a82"
server_key_iv: "bc5d7a309c704743138a387be3754981"
server_key_version: "encryptionkey@0"
I20260928 08:57:51.130062 13415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "10f6a30a72ba48e596737a30cfe91a71"
format_stamp: "Formatted at 2026-09-28 08:57:51 on dist-test-slave-pbjp"
server_key: "04928ff3d8056a9db9b4fbda92ae8a82"
server_key_iv: "bc5d7a309c704743138a387be3754981"
server_key_version: "encryptionkey@0"
I20260928 08:57:51.131294 13415 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:57:51.132032 13430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:51.132184 13415 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:57:51.132268 13415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "10f6a30a72ba48e596737a30cfe91a71"
format_stamp: "Formatted at 2026-09-28 08:57:51 on dist-test-slave-pbjp"
server_key: "04928ff3d8056a9db9b4fbda92ae8a82"
server_key_iv: "bc5d7a309c704743138a387be3754981"
server_key_version: "encryptionkey@0"
I20260928 08:57:51.132328 13415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:51.144721 13415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:51.146970 13415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:51.147114 13415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:51.151263 13415 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:46581
I20260928 08:57:51.151312 13482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:46581 every 8 connection(s)
I20260928 08:57:51.151662 13415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:57:51.152618 13483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:51.155119 13483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10f6a30a72ba48e596737a30cfe91a71: Bootstrap starting.
I20260928 08:57:51.155712 13483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10f6a30a72ba48e596737a30cfe91a71: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:51.155965 13483 log.cc:824] T 00000000000000000000000000000000 P 10f6a30a72ba48e596737a30cfe91a71: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:51.156725 13483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10f6a30a72ba48e596737a30cfe91a71: No bootstrap required, opened a new log
I20260928 08:57:51.158337 13483 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10f6a30a72ba48e596737a30cfe91a71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10f6a30a72ba48e596737a30cfe91a71" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46581 } }
I20260928 08:57:51.158494 13483 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10f6a30a72ba48e596737a30cfe91a71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:51.158516 13483 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10f6a30a72ba48e596737a30cfe91a71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f6a30a72ba48e596737a30cfe91a71, State: Initialized, Role: FOLLOWER
I20260928 08:57:51.158650 13483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10f6a30a72ba48e596737a30cfe91a71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10f6a30a72ba48e596737a30cfe91a71" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46581 } }
I20260928 08:57:51.158715 13483 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10f6a30a72ba48e596737a30cfe91a71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:57:51.158736 13483 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10f6a30a72ba48e596737a30cfe91a71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:57:51.158766 13483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10f6a30a72ba48e596737a30cfe91a71 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:51.159610 13483 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10f6a30a72ba48e596737a30cfe91a71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10f6a30a72ba48e596737a30cfe91a71" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46581 } }
I20260928 08:57:51.159699 13483 leader_election.cc:302] T 00000000000000000000000000000000 P 10f6a30a72ba48e596737a30cfe91a71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10f6a30a72ba48e596737a30cfe91a71; no voters:
I20260928 08:57:51.159853 13483 leader_election.cc:288] T 00000000000000000000000000000000 P 10f6a30a72ba48e596737a30cfe91a71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:57:51.159900 13486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10f6a30a72ba48e596737a30cfe91a71 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:57:51.160054 13486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10f6a30a72ba48e596737a30cfe91a71 [term 1 LEADER]: Becoming Leader. State: Replica: 10f6a30a72ba48e596737a30cfe91a71, State: Running, Role: LEADER
I20260928 08:57:51.160109 13483 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10f6a30a72ba48e596737a30cfe91a71 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:57:51.160171 13486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10f6a30a72ba48e596737a30cfe91a71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10f6a30a72ba48e596737a30cfe91a71" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46581 } }
I20260928 08:57:51.160537 13487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10f6a30a72ba48e596737a30cfe91a71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10f6a30a72ba48e596737a30cfe91a71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10f6a30a72ba48e596737a30cfe91a71" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46581 } } }
I20260928 08:57:51.160652 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 13415
I20260928 08:57:51.160758 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:57:51.160962 24223 external_mini_cluster.cc:1428] Setting key 2eb8a5d9f22f40b7939ed1f0b884a0a8
I20260928 08:57:51.161259 13487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10f6a30a72ba48e596737a30cfe91a71 [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:51.161945 13488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10f6a30a72ba48e596737a30cfe91a71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10f6a30a72ba48e596737a30cfe91a71. Latest consensus state: current_term: 1 leader_uuid: "10f6a30a72ba48e596737a30cfe91a71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10f6a30a72ba48e596737a30cfe91a71" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 46581 } } }
I20260928 08:57:51.162045 13488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10f6a30a72ba48e596737a30cfe91a71 [sys.catalog]: This master's current role is: LEADER
W20260928 08:57:51.162650 13503 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 10f6a30a72ba48e596737a30cfe91a71: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:57:51.162921 13503 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:57:51.163069 13497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:57:51.165453 13497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:57:51.167090 13497 catalog_manager.cc:1347] Generated new cluster ID: 8ab67238b4394c21a4b8b31d14443745
I20260928 08:57:51.167135 13497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:57:51.169396 13506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:51.162716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52473 (local address 127.23.167.254:46581)
0928 08:57:51.163050 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:51.163061 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:51.163096 (+ 35us) server_negotiation.cc:407] Connection header received
0928 08:57:51.163313 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:51.163321 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:51.163414 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:51.163685 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:51.163870 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:51.165490 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:51.166719 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:51.167460 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:51.168330 (+ 870us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:51.168624 (+ 294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:51.168627 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:51.168628 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:51.168811 (+ 183us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:51.168853 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:51.168867 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:51.169106 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:51.169150 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 08:57:51.169181 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":69,"threads_started":1}
I20260928 08:57:51.170374 24223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:57:51.175755 13506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:51.171813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52775 (local address 127.23.167.254:46581)
0928 08:57:51.171855 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:51.171857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:51.171945 (+ 88us) server_negotiation.cc:407] Connection header received
0928 08:57:51.172018 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:51.172019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:51.172029 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:51.172074 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:51.172245 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:51.173140 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:51.174283 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:51.174590 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:51.174623 (+ 33us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:51.175351 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:51.175352 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:51.175352 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:51.175357 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:51.175372 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:51.175377 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:51.175405 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:51.175662 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 08:57:51.175675 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 08:57:51.177404 13497 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:57:51.178086 13497 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:57:51.192447 13497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10f6a30a72ba48e596737a30cfe91a71: Generated new TSK 0
I20260928 08:57:51.192782 13497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:57:51.195950 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 13415
W20260928 08:57:51.200807 13400 connection.cc:535] client connection to 127.23.167.254:46581 recv error: Network error: failed to read from TLS socket (remote: 127.23.167.254:46581): Connection reset by peer (error 104)
2026-09-28T08:57:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1222 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:57:51 dist-test-slave-pbjp krb5kdc[13515](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:57:51 dist-test-slave-pbjp krb5kdc[13515](info): set up 2 sockets
Sep 28 08:57:51 dist-test-slave-pbjp krb5kdc[13515](info): commencing operation
krb5kdc: starting...
W20260928 08:57:52.144982 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:57:52 dist-test-slave-pbjp krb5kdc[13515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:57:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:57:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.167.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
I20260928 08:57:52.258265 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:43891
--webserver_interface=127.23.167.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:41173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:43891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:52.338903 13528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:52.339119 13528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:52.339141 13528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:52.340261 13528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:57:52.340298 13528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:52.340312 13528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:52.340322 13528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:57:52.340332 13528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:57:52.341841 13528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:41173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:43891
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:43891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:52.342038 13528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:52.342216 13528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:52.347137 13536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:52.347141 13533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:52.347142 13534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:52.347391 13528 server_base.cc:1034] running on GCE node
I20260928 08:57:52.347683 13528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:52.347982 13528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:52.349119 13528 hybrid_clock.cc:648] HybridClock initialized: now 1790585872349105 us; error 30 us; skew 500 ppm
Sep 28 08:57:52 dist-test-slave-pbjp krb5kdc[13515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585872, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:52.351472 13528 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:57:52.351893 13528 webserver.cc:460] Webserver started at http://127.23.167.254:38433/ using document root <none> and password file <none>
I20260928 08:57:52.352095 13528 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:52.352142 13528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:52.352262 13528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:52.353389 13528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "7e0a8c35ffd0415aba12e94ef45b33ad"
format_stamp: "Formatted at 2026-09-28 08:57:52 on dist-test-slave-pbjp"
server_key: "ab48ad0a3d8ff68ba67b2ec6a852d303"
server_key_iv: "7e6ee31ee82945a904e219d3a8707cb6"
server_key_version: "encryptionkey@0"
I20260928 08:57:52.353713 13528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "7e0a8c35ffd0415aba12e94ef45b33ad"
format_stamp: "Formatted at 2026-09-28 08:57:52 on dist-test-slave-pbjp"
server_key: "ab48ad0a3d8ff68ba67b2ec6a852d303"
server_key_iv: "7e6ee31ee82945a904e219d3a8707cb6"
server_key_version: "encryptionkey@0"
I20260928 08:57:52.354970 13528 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:57:52.355677 13543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:52.355808 13528 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:57:52.355890 13528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "7e0a8c35ffd0415aba12e94ef45b33ad"
format_stamp: "Formatted at 2026-09-28 08:57:52 on dist-test-slave-pbjp"
server_key: "ab48ad0a3d8ff68ba67b2ec6a852d303"
server_key_iv: "7e6ee31ee82945a904e219d3a8707cb6"
server_key_version: "encryptionkey@0"
I20260928 08:57:52.355963 13528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:52.359261 13528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:52.361233 13528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:52.361361 13528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:52.364961 13528 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:43891
I20260928 08:57:52.364987 13595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:43891 every 8 connection(s)
I20260928 08:57:52.365322 13528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:57:52.366267 13596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:52.368549 13596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e0a8c35ffd0415aba12e94ef45b33ad: Bootstrap starting.
I20260928 08:57:52.369098 13596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e0a8c35ffd0415aba12e94ef45b33ad: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:52.369335 13596 log.cc:824] T 00000000000000000000000000000000 P 7e0a8c35ffd0415aba12e94ef45b33ad: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:52.369936 13596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e0a8c35ffd0415aba12e94ef45b33ad: No bootstrap required, opened a new log
I20260928 08:57:52.371474 13596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e0a8c35ffd0415aba12e94ef45b33ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0a8c35ffd0415aba12e94ef45b33ad" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43891 } }
I20260928 08:57:52.371608 13596 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e0a8c35ffd0415aba12e94ef45b33ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:52.371627 13596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e0a8c35ffd0415aba12e94ef45b33ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e0a8c35ffd0415aba12e94ef45b33ad, State: Initialized, Role: FOLLOWER
I20260928 08:57:52.371737 13596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e0a8c35ffd0415aba12e94ef45b33ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0a8c35ffd0415aba12e94ef45b33ad" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43891 } }
I20260928 08:57:52.371806 13596 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e0a8c35ffd0415aba12e94ef45b33ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:57:52.371834 13596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e0a8c35ffd0415aba12e94ef45b33ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:57:52.371866 13596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e0a8c35ffd0415aba12e94ef45b33ad [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:52.372633 13596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e0a8c35ffd0415aba12e94ef45b33ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0a8c35ffd0415aba12e94ef45b33ad" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43891 } }
I20260928 08:57:52.372720 13596 leader_election.cc:302] T 00000000000000000000000000000000 P 7e0a8c35ffd0415aba12e94ef45b33ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e0a8c35ffd0415aba12e94ef45b33ad; no voters:
I20260928 08:57:52.372829 13596 leader_election.cc:288] T 00000000000000000000000000000000 P 7e0a8c35ffd0415aba12e94ef45b33ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:57:52.372870 13599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e0a8c35ffd0415aba12e94ef45b33ad [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:57:52.373015 13599 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e0a8c35ffd0415aba12e94ef45b33ad [term 1 LEADER]: Becoming Leader. State: Replica: 7e0a8c35ffd0415aba12e94ef45b33ad, State: Running, Role: LEADER
I20260928 08:57:52.373100 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 13528
I20260928 08:57:52.373056 13596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e0a8c35ffd0415aba12e94ef45b33ad [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:57:52.373198 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:57:52.373217 13599 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e0a8c35ffd0415aba12e94ef45b33ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0a8c35ffd0415aba12e94ef45b33ad" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43891 } }
I20260928 08:57:52.373411 24223 external_mini_cluster.cc:1428] Setting key 8162872017a5dca18c5104ec8278f929
I20260928 08:57:52.373622 13601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e0a8c35ffd0415aba12e94ef45b33ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e0a8c35ffd0415aba12e94ef45b33ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0a8c35ffd0415aba12e94ef45b33ad" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43891 } } }
I20260928 08:57:52.373694 13601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e0a8c35ffd0415aba12e94ef45b33ad [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:52.373867 13607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:57:52.374356 13607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:57:52.374547 13600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e0a8c35ffd0415aba12e94ef45b33ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e0a8c35ffd0415aba12e94ef45b33ad. Latest consensus state: current_term: 1 leader_uuid: "7e0a8c35ffd0415aba12e94ef45b33ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0a8c35ffd0415aba12e94ef45b33ad" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43891 } } }
I20260928 08:57:52.374725 13600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e0a8c35ffd0415aba12e94ef45b33ad [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:52.376276 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:52.374020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51909 (local address 127.23.167.254:43891)
0928 08:57:52.374835 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:52.374845 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:52.374875 (+ 30us) server_negotiation.cc:407] Connection header received
0928 08:57:52.374989 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:52.374996 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:52.375039 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:52.375174 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:52.375846 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:52.375849 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:52.375849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:52.375854 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:52.375894 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:52.375907 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:52.375974 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:52.376053 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 08:57:52.376098 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":74,"threads_started":1}
I20260928 08:57:52.376394 13607 catalog_manager.cc:1347] Generated new cluster ID: 15e62158af6e43379d45837d52aceb5f
I20260928 08:57:52.376464 13607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:57:52.377133 24223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:57:52.379534 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:52.378461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60021 (local address 127.23.167.254:43891)
0928 08:57:52.378494 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:52.378495 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:52.378674 (+ 179us) server_negotiation.cc:407] Connection header received
0928 08:57:52.378737 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:52.378738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:52.378744 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:52.378794 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:52.379284 (+ 490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:52.379285 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:52.379286 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:52.379290 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:52.379314 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:52.379318 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:52.379404 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:52.379462 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 08:57:52.379474 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 08:57:52.389053 13607 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:57:52.389225 13607 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:57:52.395372 13607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e0a8c35ffd0415aba12e94ef45b33ad: Generated new TSK 0
I20260928 08:57:52.395560 13607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:57:52.402087 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 13528
2026-09-28T08:57:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1204 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:57:52 dist-test-slave-pbjp krb5kdc[13627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:57:52 dist-test-slave-pbjp krb5kdc[13627](info): set up 2 sockets
Sep 28 08:57:52 dist-test-slave-pbjp krb5kdc[13627](info): commencing operation
krb5kdc: starting...
W20260928 08:57:53.356923 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:57:53 dist-test-slave-pbjp krb5kdc[13627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:57:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:57:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.167.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
I20260928 08:57:53.468034 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:34827
--webserver_interface=127.23.167.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:37235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:34827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:53.547353 13640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:53.547545 13640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:53.547566 13640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:53.548648 13640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:57:53.548681 13640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:53.548693 13640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:53.548704 13640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:57:53.548715 13640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:57:53.550141 13640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:37235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:34827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:34827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:53.550343 13640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:53.550515 13640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:53.555081 13646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:53.555168 13640 server_base.cc:1034] running on GCE node
W20260928 08:57:53.555079 13648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:53.555255 13645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:53.555550 13640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:53.555809 13640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:53.557123 13640 hybrid_clock.cc:648] HybridClock initialized: now 1790585873556965 us; error 31 us; skew 500 ppm
Sep 28 08:57:53 dist-test-slave-pbjp krb5kdc[13627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585873, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:53.559429 13640 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:57:53.559854 13640 webserver.cc:460] Webserver started at http://127.23.167.254:44579/ using document root <none> and password file <none>
I20260928 08:57:53.560045 13640 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:53.560091 13640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:53.560211 13640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:53.561275 13640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "d2d2bb25224746d9bda813dd629aedb8"
format_stamp: "Formatted at 2026-09-28 08:57:53 on dist-test-slave-pbjp"
server_key: "1588e2cfa0e5f64bbc01f79a99d00278"
server_key_iv: "4e6705181d5e413838aa14e61a18d155"
server_key_version: "encryptionkey@0"
I20260928 08:57:53.561604 13640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "d2d2bb25224746d9bda813dd629aedb8"
format_stamp: "Formatted at 2026-09-28 08:57:53 on dist-test-slave-pbjp"
server_key: "1588e2cfa0e5f64bbc01f79a99d00278"
server_key_iv: "4e6705181d5e413838aa14e61a18d155"
server_key_version: "encryptionkey@0"
I20260928 08:57:53.562897 13640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:57:53.563542 13655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:53.563701 13640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:57:53.563777 13640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "d2d2bb25224746d9bda813dd629aedb8"
format_stamp: "Formatted at 2026-09-28 08:57:53 on dist-test-slave-pbjp"
server_key: "1588e2cfa0e5f64bbc01f79a99d00278"
server_key_iv: "4e6705181d5e413838aa14e61a18d155"
server_key_version: "encryptionkey@0"
I20260928 08:57:53.563828 13640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:53.609099 13640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:53.611186 13640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:53.611305 13640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:53.615123 13640 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:34827
I20260928 08:57:53.615162 13707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:34827 every 8 connection(s)
I20260928 08:57:53.615495 13640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:57:53.616402 13708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:53.618726 13708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2d2bb25224746d9bda813dd629aedb8: Bootstrap starting.
I20260928 08:57:53.619323 13708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2d2bb25224746d9bda813dd629aedb8: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:53.619578 13708 log.cc:824] T 00000000000000000000000000000000 P d2d2bb25224746d9bda813dd629aedb8: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:53.620283 13708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2d2bb25224746d9bda813dd629aedb8: No bootstrap required, opened a new log
I20260928 08:57:53.621994 13708 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2d2bb25224746d9bda813dd629aedb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2d2bb25224746d9bda813dd629aedb8" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34827 } }
I20260928 08:57:53.622138 13708 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2d2bb25224746d9bda813dd629aedb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:53.622159 13708 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2d2bb25224746d9bda813dd629aedb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d2bb25224746d9bda813dd629aedb8, State: Initialized, Role: FOLLOWER
I20260928 08:57:53.622308 13708 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2d2bb25224746d9bda813dd629aedb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2d2bb25224746d9bda813dd629aedb8" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34827 } }
I20260928 08:57:53.622385 13708 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2d2bb25224746d9bda813dd629aedb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:57:53.622447 13708 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2d2bb25224746d9bda813dd629aedb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:57:53.622496 13708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2d2bb25224746d9bda813dd629aedb8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:53.622897 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 13640
I20260928 08:57:53.622997 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:57:53.623152 24223 external_mini_cluster.cc:1428] Setting key 3fa2c8e58acfdc61962bddb0b3fa2852
I20260928 08:57:53.623476 13708 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2d2bb25224746d9bda813dd629aedb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2d2bb25224746d9bda813dd629aedb8" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34827 } }
I20260928 08:57:53.623631 13708 leader_election.cc:302] T 00000000000000000000000000000000 P d2d2bb25224746d9bda813dd629aedb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2d2bb25224746d9bda813dd629aedb8; no voters:
I20260928 08:57:53.623803 13708 leader_election.cc:288] T 00000000000000000000000000000000 P d2d2bb25224746d9bda813dd629aedb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:57:53.623823 13713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2d2bb25224746d9bda813dd629aedb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:57:53.623981 13713 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2d2bb25224746d9bda813dd629aedb8 [term 1 LEADER]: Becoming Leader. State: Replica: d2d2bb25224746d9bda813dd629aedb8, State: Running, Role: LEADER
I20260928 08:57:53.624099 13708 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2d2bb25224746d9bda813dd629aedb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:57:53.624073 13713 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2d2bb25224746d9bda813dd629aedb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2d2bb25224746d9bda813dd629aedb8" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34827 } }
I20260928 08:57:53.624390 13713 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2d2bb25224746d9bda813dd629aedb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2d2bb25224746d9bda813dd629aedb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2d2bb25224746d9bda813dd629aedb8" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34827 } } }
I20260928 08:57:53.624519 13713 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2d2bb25224746d9bda813dd629aedb8 [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:53.624689 13713 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2d2bb25224746d9bda813dd629aedb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2d2bb25224746d9bda813dd629aedb8. Latest consensus state: current_term: 1 leader_uuid: "d2d2bb25224746d9bda813dd629aedb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2d2bb25224746d9bda813dd629aedb8" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 34827 } } }
I20260928 08:57:53.624794 13713 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2d2bb25224746d9bda813dd629aedb8 [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:53.625131 13720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:57:53.625676 13720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:57:53.627892 13720 catalog_manager.cc:1347] Generated new cluster ID: b88696abaec84310ad0d6273cd0fbc04
I20260928 08:57:53.627945 13720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 08:57:53 dist-test-slave-pbjp krb5kdc[13627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:57:53.633447 13712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:53.623600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47259 (local address 127.23.167.254:34827)
0928 08:57:53.623824 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:53.623834 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:53.623871 (+ 37us) server_negotiation.cc:407] Connection header received
0928 08:57:53.624008 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:53.624016 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:53.624073 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:53.624213 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:53.625141 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:53.626485 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:53.627536 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:53.628064 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:53.628159 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:53.630396 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:53.630400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:53.630400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:53.630413 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:53.631939 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:53.632277 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:53.632280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:53.632280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:53.632379 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:53.632556 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:53.632557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:53.632557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:53.632809 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:53.633022 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:53.633116 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 08:57:53.633268 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":90,"threads_started":1}
I20260928 08:57:53.634356 24223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:57:53.639220 13720 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:57:53.640100 13720 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:57:53.643062 13712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:53.635610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34649 (local address 127.23.167.254:34827)
0928 08:57:53.635645 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:53.635647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:53.637310 (+ 1663us) server_negotiation.cc:407] Connection header received
0928 08:57:53.637368 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:53.637369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:53.637382 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:53.637430 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:53.637941 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:53.638422 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:53.639389 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:53.639607 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:53.639663 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:53.640792 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:53.640793 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:53.640793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:53.640800 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:53.642081 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:53.642381 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:53.642382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:53.642382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:53.642432 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:53.642597 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:53.642598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:53.642598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:53.642697 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:53.642778 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:53.642883 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 08:57:53.642968 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 08:57:53.644953 13720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2d2bb25224746d9bda813dd629aedb8: Generated new TSK 0
I20260928 08:57:53.645159 13720 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:57:53.648900 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 13640
2026-09-28T08:57:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1247 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:57:53 dist-test-slave-pbjp krb5kdc[13739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:57:53 dist-test-slave-pbjp krb5kdc[13739](info): set up 2 sockets
Sep 28 08:57:53 dist-test-slave-pbjp krb5kdc[13739](info): commencing operation
krb5kdc: starting...
W20260928 08:57:54.596994 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:57:54 dist-test-slave-pbjp krb5kdc[13739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:57:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:57:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.167.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
I20260928 08:57:54.708155 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:39487
--webserver_interface=127.23.167.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:39487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:54.787228 13752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:54.787418 13752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:54.787442 13752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:54.788537 13752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:57:54.788575 13752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:54.788589 13752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:54.788616 13752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:57:54.788636 13752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:57:54.790113 13752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:39487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:39487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:54.790318 13752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:54.790521 13752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 08:57:54.795050 13752 server_base.cc:1034] running on GCE node
W20260928 08:57:54.795087 13760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:54.794986 13757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:54.794976 13758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:54.795419 13752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:54.795686 13752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:54.796818 13752 hybrid_clock.cc:648] HybridClock initialized: now 1790585874796811 us; error 27 us; skew 500 ppm
Sep 28 08:57:54 dist-test-slave-pbjp krb5kdc[13739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585874, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:54.802166 13752 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:57:54.802605 13752 webserver.cc:460] Webserver started at http://127.23.167.254:39879/ using document root <none> and password file <none>
I20260928 08:57:54.802812 13752 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:54.802858 13752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:54.802982 13752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:54.803993 13752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "8a46358383d048bbb81a5a090db149db"
format_stamp: "Formatted at 2026-09-28 08:57:54 on dist-test-slave-pbjp"
server_key: "d33849fbef6b0ced6ea97c9e2a3ed30d"
server_key_iv: "c92517410b4c70ef315c6aa7dea4933e"
server_key_version: "encryptionkey@0"
I20260928 08:57:54.804322 13752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "8a46358383d048bbb81a5a090db149db"
format_stamp: "Formatted at 2026-09-28 08:57:54 on dist-test-slave-pbjp"
server_key: "d33849fbef6b0ced6ea97c9e2a3ed30d"
server_key_iv: "c92517410b4c70ef315c6aa7dea4933e"
server_key_version: "encryptionkey@0"
I20260928 08:57:54.805609 13752 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 08:57:54.806330 13767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:54.806483 13752 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:57:54.806564 13752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "8a46358383d048bbb81a5a090db149db"
format_stamp: "Formatted at 2026-09-28 08:57:54 on dist-test-slave-pbjp"
server_key: "d33849fbef6b0ced6ea97c9e2a3ed30d"
server_key_iv: "c92517410b4c70ef315c6aa7dea4933e"
server_key_version: "encryptionkey@0"
I20260928 08:57:54.806623 13752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:54.826092 13752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:54.827998 13752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:54.828146 13752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:54.832190 13752 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:39487
I20260928 08:57:54.832224 13819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:39487 every 8 connection(s)
I20260928 08:57:54.832556 13752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:57:54.832693 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 13752
I20260928 08:57:54.832803 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:57:54.832999 24223 external_mini_cluster.cc:1428] Setting key f91263d1c54126c7448356b40014f927
I20260928 08:57:54.833564 13820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:54.836550 13820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a46358383d048bbb81a5a090db149db: Bootstrap starting.
I20260928 08:57:54.837186 13820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a46358383d048bbb81a5a090db149db: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:54.837500 13820 log.cc:824] T 00000000000000000000000000000000 P 8a46358383d048bbb81a5a090db149db: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:54.838197 13820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a46358383d048bbb81a5a090db149db: No bootstrap required, opened a new log
I20260928 08:57:54.838320 13823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:54.833533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34517 (local address 127.23.167.254:39487)
0928 08:57:54.833809 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:54.833825 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:54.833852 (+ 27us) server_negotiation.cc:407] Connection header received
0928 08:57:54.834267 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:54.834273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:54.834321 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:54.834434 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:54.834689 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:54.835570 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:54.836570 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:54.837016 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:54.837133 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:54.837692 (+ 559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:54.837695 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:54.837696 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:54.837866 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:54.837896 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:54.837906 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:54.837954 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:54.838046 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 08:57:54.838095 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":71,"threads_started":1}
I20260928 08:57:54.839180 24223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:57:54.840209 13820 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a46358383d048bbb81a5a090db149db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a46358383d048bbb81a5a090db149db" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 39487 } }
I20260928 08:57:54.840342 13820 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a46358383d048bbb81a5a090db149db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:54.840376 13820 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a46358383d048bbb81a5a090db149db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a46358383d048bbb81a5a090db149db, State: Initialized, Role: FOLLOWER
I20260928 08:57:54.840538 13820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a46358383d048bbb81a5a090db149db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a46358383d048bbb81a5a090db149db" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 39487 } }
I20260928 08:57:54.840624 13820 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a46358383d048bbb81a5a090db149db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:57:54.840655 13820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a46358383d048bbb81a5a090db149db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:57:54.840704 13820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a46358383d048bbb81a5a090db149db [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:54.841390 13820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a46358383d048bbb81a5a090db149db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a46358383d048bbb81a5a090db149db" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 39487 } }
I20260928 08:57:54.841497 13820 leader_election.cc:302] T 00000000000000000000000000000000 P 8a46358383d048bbb81a5a090db149db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a46358383d048bbb81a5a090db149db; no voters:
I20260928 08:57:54.841655 13820 leader_election.cc:288] T 00000000000000000000000000000000 P 8a46358383d048bbb81a5a090db149db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:57:54.841735 13830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a46358383d048bbb81a5a090db149db [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:57:54.841900 13820 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a46358383d048bbb81a5a090db149db [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:57:54.841938 13830 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a46358383d048bbb81a5a090db149db [term 1 LEADER]: Becoming Leader. State: Replica: 8a46358383d048bbb81a5a090db149db, State: Running, Role: LEADER
I20260928 08:57:54.842123 13830 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a46358383d048bbb81a5a090db149db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a46358383d048bbb81a5a090db149db" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 39487 } }
I20260928 08:57:54.842440 13831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a46358383d048bbb81a5a090db149db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a46358383d048bbb81a5a090db149db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a46358383d048bbb81a5a090db149db" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 39487 } } }
I20260928 08:57:54.842520 13831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a46358383d048bbb81a5a090db149db [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:54.842670 13832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a46358383d048bbb81a5a090db149db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a46358383d048bbb81a5a090db149db. Latest consensus state: current_term: 1 leader_uuid: "8a46358383d048bbb81a5a090db149db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a46358383d048bbb81a5a090db149db" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 39487 } } }
I20260928 08:57:54.842737 13832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a46358383d048bbb81a5a090db149db [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:54.843125 13839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:57:54.843667 13839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:57:54.843775 13823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:54.840407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38221 (local address 127.23.167.254:39487)
0928 08:57:54.840443 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:54.840445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:54.840564 (+ 119us) server_negotiation.cc:407] Connection header received
0928 08:57:54.840606 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:54.840606 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:54.840616 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:54.840674 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:54.840826 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:54.841532 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:54.842484 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:54.842777 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:54.842817 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:54.843542 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:54.843543 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:54.843543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:54.843548 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:54.843565 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:54.843569 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:54.843630 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:54.843689 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 08:57:54.843713 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 08:57:54.845134 13839 catalog_manager.cc:1347] Generated new cluster ID: faeb229d87364ecd9b0f66ac6af4d437
I20260928 08:57:54.845191 13839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:57:54.851886 13839 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:57:54.852377 13839 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:57:54.858260 13839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a46358383d048bbb81a5a090db149db: Generated new TSK 0
I20260928 08:57:54.858423 13839 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:57:54.866969 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 13752
2026-09-28T08:57:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1217 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:57:54 dist-test-slave-pbjp krb5kdc[13851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:57:54 dist-test-slave-pbjp krb5kdc[13851](info): set up 2 sockets
Sep 28 08:57:54 dist-test-slave-pbjp krb5kdc[13851](info): commencing operation
krb5kdc: starting...
W20260928 08:57:55.813022 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:57:55 dist-test-slave-pbjp krb5kdc[13851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:57:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:57:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.167.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
I20260928 08:57:55.923959 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:42473
--webserver_interface=127.23.167.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:42299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:42473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:56.002228 13864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:56.002437 13864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:56.002475 13864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:56.003613 13864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:57:56.003659 13864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:56.003677 13864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:56.003693 13864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:57:56.003708 13864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:57:56.005259 13864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:42299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:42473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:42473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:56.005519 13864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:56.005733 13864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:56.010587 13872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:56.010578 13870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:56.010609 13864 server_base.cc:1034] running on GCE node
W20260928 08:57:56.010578 13869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:56.011091 13864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:56.011384 13864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:56.012557 13864 hybrid_clock.cc:648] HybridClock initialized: now 1790585876012540 us; error 27 us; skew 500 ppm
Sep 28 08:57:56 dist-test-slave-pbjp krb5kdc[13851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585876, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:56.015156 13864 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:57:56.015616 13864 webserver.cc:460] Webserver started at http://127.23.167.254:43681/ using document root <none> and password file <none>
I20260928 08:57:56.015820 13864 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:56.015867 13864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:56.016001 13864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:56.017095 13864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "a55cad777af8493aad094ce4b2fa9cf6"
format_stamp: "Formatted at 2026-09-28 08:57:56 on dist-test-slave-pbjp"
server_key: "2dd1c71999d8f5b2ce34d8da92183a78"
server_key_iv: "822ff5e47b5a17454ed2dd08e121a854"
server_key_version: "encryptionkey@0"
I20260928 08:57:56.017452 13864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "a55cad777af8493aad094ce4b2fa9cf6"
format_stamp: "Formatted at 2026-09-28 08:57:56 on dist-test-slave-pbjp"
server_key: "2dd1c71999d8f5b2ce34d8da92183a78"
server_key_iv: "822ff5e47b5a17454ed2dd08e121a854"
server_key_version: "encryptionkey@0"
I20260928 08:57:56.018718 13864 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:57:56.019433 13879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:56.019608 13864 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:57:56.019692 13864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "a55cad777af8493aad094ce4b2fa9cf6"
format_stamp: "Formatted at 2026-09-28 08:57:56 on dist-test-slave-pbjp"
server_key: "2dd1c71999d8f5b2ce34d8da92183a78"
server_key_iv: "822ff5e47b5a17454ed2dd08e121a854"
server_key_version: "encryptionkey@0"
I20260928 08:57:56.019757 13864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:56.023080 13864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:56.024777 13864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:56.024902 13864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:56.028447 13864 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:42473
I20260928 08:57:56.028487 13931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:42473 every 8 connection(s)
I20260928 08:57:56.028777 13864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:57:56.029743 13932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:56.031914 13932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a55cad777af8493aad094ce4b2fa9cf6: Bootstrap starting.
I20260928 08:57:56.032428 13932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a55cad777af8493aad094ce4b2fa9cf6: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:56.032662 13932 log.cc:824] T 00000000000000000000000000000000 P a55cad777af8493aad094ce4b2fa9cf6: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:56.033290 13932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a55cad777af8493aad094ce4b2fa9cf6: No bootstrap required, opened a new log
I20260928 08:57:56.034863 13932 raft_consensus.cc:348] T 00000000000000000000000000000000 P a55cad777af8493aad094ce4b2fa9cf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a55cad777af8493aad094ce4b2fa9cf6" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42473 } }
I20260928 08:57:56.034999 13932 raft_consensus.cc:374] T 00000000000000000000000000000000 P a55cad777af8493aad094ce4b2fa9cf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:57:56.035020 13932 raft_consensus.cc:729] T 00000000000000000000000000000000 P a55cad777af8493aad094ce4b2fa9cf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a55cad777af8493aad094ce4b2fa9cf6, State: Initialized, Role: FOLLOWER
I20260928 08:57:56.035132 13932 consensus_queue.cc:260] T 00000000000000000000000000000000 P a55cad777af8493aad094ce4b2fa9cf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a55cad777af8493aad094ce4b2fa9cf6" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42473 } }
I20260928 08:57:56.035204 13932 raft_consensus.cc:388] T 00000000000000000000000000000000 P a55cad777af8493aad094ce4b2fa9cf6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:57:56.035233 13932 raft_consensus.cc:482] T 00000000000000000000000000000000 P a55cad777af8493aad094ce4b2fa9cf6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:57:56.035264 13932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a55cad777af8493aad094ce4b2fa9cf6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:56.036059 13932 raft_consensus.cc:504] T 00000000000000000000000000000000 P a55cad777af8493aad094ce4b2fa9cf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a55cad777af8493aad094ce4b2fa9cf6" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42473 } }
I20260928 08:57:56.036149 13932 leader_election.cc:302] T 00000000000000000000000000000000 P a55cad777af8493aad094ce4b2fa9cf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a55cad777af8493aad094ce4b2fa9cf6; no voters:
I20260928 08:57:56.036280 13932 leader_election.cc:288] T 00000000000000000000000000000000 P a55cad777af8493aad094ce4b2fa9cf6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:57:56.036324 13935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a55cad777af8493aad094ce4b2fa9cf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:57:56.036466 13935 raft_consensus.cc:686] T 00000000000000000000000000000000 P a55cad777af8493aad094ce4b2fa9cf6 [term 1 LEADER]: Becoming Leader. State: Replica: a55cad777af8493aad094ce4b2fa9cf6, State: Running, Role: LEADER
I20260928 08:57:56.036523 13932 sys_catalog.cc:564] T 00000000000000000000000000000000 P a55cad777af8493aad094ce4b2fa9cf6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:57:56.036563 13935 consensus_queue.cc:237] T 00000000000000000000000000000000 P a55cad777af8493aad094ce4b2fa9cf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a55cad777af8493aad094ce4b2fa9cf6" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42473 } }
I20260928 08:57:56.036898 13936 sys_catalog.cc:455] T 00000000000000000000000000000000 P a55cad777af8493aad094ce4b2fa9cf6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a55cad777af8493aad094ce4b2fa9cf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a55cad777af8493aad094ce4b2fa9cf6" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42473 } } }
I20260928 08:57:56.036902 13937 sys_catalog.cc:455] T 00000000000000000000000000000000 P a55cad777af8493aad094ce4b2fa9cf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a55cad777af8493aad094ce4b2fa9cf6. Latest consensus state: current_term: 1 leader_uuid: "a55cad777af8493aad094ce4b2fa9cf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a55cad777af8493aad094ce4b2fa9cf6" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 42473 } } }
I20260928 08:57:56.037082 13937 sys_catalog.cc:458] T 00000000000000000000000000000000 P a55cad777af8493aad094ce4b2fa9cf6 [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:56.037089 13936 sys_catalog.cc:458] T 00000000000000000000000000000000 P a55cad777af8493aad094ce4b2fa9cf6 [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:56.037478 13944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:57:56.038010 13944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:57:56.038357 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 13864
I20260928 08:57:56.038453 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:57:56.038615 24223 external_mini_cluster.cc:1428] Setting key 07fbed33b3f2df98e41ef2f0b8321052
I20260928 08:57:56.040302 13944 catalog_manager.cc:1347] Generated new cluster ID: a62ed9ab6df14dba9779923adc81aab8
I20260928 08:57:56.040351 13944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:57:56.041733 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:56.039227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39373 (local address 127.23.167.254:42473)
0928 08:57:56.039402 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:56.039412 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:56.039445 (+ 33us) server_negotiation.cc:407] Connection header received
0928 08:57:56.039567 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:56.039574 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:56.039605 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:56.039749 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:56.041307 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:56.041311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:56.041311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:56.041316 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:56.041364 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:56.041381 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:56.041449 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:56.041511 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 08:57:56.041556 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260928 08:57:56.046022 24223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:57:56.048626 13944 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:57:56.048792 13944 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:57:56.048843 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:56.047610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41583 (local address 127.23.167.254:42473)
0928 08:57:56.047703 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:56.047705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:56.047754 (+ 49us) server_negotiation.cc:407] Connection header received
0928 08:57:56.047836 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:56.047837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:56.047844 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:56.047893 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:56.048597 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:56.048598 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:56.048598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:57:56.048603 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:56.048620 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:57:56.048625 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:56.048674 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:56.048741 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 08:57:56.048756 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 08:57:56.057466 13944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a55cad777af8493aad094ce4b2fa9cf6: Generated new TSK 0
I20260928 08:57:56.057654 13944 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:57:56.063313 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 13864
2026-09-28T08:57:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1196 ms)
[----------] 18 tests from AuthTokenIssuingTest (21881 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:57:56 dist-test-slave-pbjp krb5kdc[13963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:57:56 dist-test-slave-pbjp krb5kdc[13963](info): set up 2 sockets
Sep 28 08:57:56 dist-test-slave-pbjp krb5kdc[13963](info): commencing operation
krb5kdc: starting...
W20260928 08:57:57.016974 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:57:57 dist-test-slave-pbjp krb5kdc[13963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:57:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:57:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.167.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
I20260928 08:57:57.127763 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:43415
--webserver_interface=127.23.167.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:37415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:43415,127.23.167.253:43757,127.23.167.252:39927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:57.206924 13976 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 08:57:57.207175 13976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:57.207198 13976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:57.207226 13976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:57.208376 13976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:57:57.208418 13976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:57.208436 13976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:57.208451 13976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:57:57.208467 13976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:57:57.209990 13976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:37415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:43415,127.23.167.253:43757,127.23.167.252:39927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:43415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:57.210246 13976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:57.210451 13976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:57.215106 13981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:57.215093 13982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:57.215121 13984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:57.215286 13976 server_base.cc:1034] running on GCE node
I20260928 08:57:57.215509 13976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:57.215775 13976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:57.216915 13976 hybrid_clock.cc:648] HybridClock initialized: now 1790585877216901 us; error 28 us; skew 500 ppm
Sep 28 08:57:57 dist-test-slave-pbjp krb5kdc[13963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585877, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:57.219192 13976 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:57:57.219595 13976 webserver.cc:460] Webserver started at http://127.23.167.254:39605/ using document root <none> and password file <none>
I20260928 08:57:57.219798 13976 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:57.219843 13976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:57.219972 13976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:57.220999 13976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "0bf5556a2e944ac99bf1c7cff58be072"
format_stamp: "Formatted at 2026-09-28 08:57:57 on dist-test-slave-pbjp"
server_key: "6fbd6f3cdffd519b576265c302618b24"
server_key_iv: "76dcd5cafe47fb8cd217fbf73830f517"
server_key_version: "encryptionkey@0"
I20260928 08:57:57.221379 13976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "0bf5556a2e944ac99bf1c7cff58be072"
format_stamp: "Formatted at 2026-09-28 08:57:57 on dist-test-slave-pbjp"
server_key: "6fbd6f3cdffd519b576265c302618b24"
server_key_iv: "76dcd5cafe47fb8cd217fbf73830f517"
server_key_version: "encryptionkey@0"
I20260928 08:57:57.222645 13976 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:57:57.223260 13991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:57.223410 13976 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:57.223487 13976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "0bf5556a2e944ac99bf1c7cff58be072"
format_stamp: "Formatted at 2026-09-28 08:57:57 on dist-test-slave-pbjp"
server_key: "6fbd6f3cdffd519b576265c302618b24"
server_key_iv: "76dcd5cafe47fb8cd217fbf73830f517"
server_key_version: "encryptionkey@0"
I20260928 08:57:57.223549 13976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:57.238047 13976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:57.239940 13976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:57.240087 13976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:57.243820 13976 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:43415
I20260928 08:57:57.243857 14043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:43415 every 8 connection(s)
I20260928 08:57:57.244199 13976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:57:57.245034 14044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:57.246634 14044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43415 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:57:57.252271 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 13976
I20260928 08:57:57.252364 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:57:57.252561 24223 external_mini_cluster.cc:1428] Setting key 45974516f5d77bb17d484fe9284ba10e
Sep 28 08:57:57 dist-test-slave-pbjp krb5kdc[13963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585877, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:57:57.256410 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.247274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:43415 (local address 127.0.0.1:41546)
0928 08:57:57.247456 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:57.247537 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:57.248036 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:57.248295 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:57.248298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:57.248583 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:57:57.248736 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:57.248741 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.249772 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.249773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:57.250440 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:57.250442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.250901 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.250903 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:57.250967 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:57.251522 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:57:57.251531 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:57:57.252988 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:57:57.254725 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:57.254733 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:57.254736 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:57.254976 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:57.255283 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:57.255284 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:57.255285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:57.255382 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:57.255818 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:57:57.255824 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:57:57.255888 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 08:57:57.255975 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 08:57:57.256144 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
I20260928 08:57:57.256419 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.247308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41546 (local address 127.23.167.254:43415)
0928 08:57:57.247467 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:57.247477 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:57.247631 (+ 154us) server_negotiation.cc:407] Connection header received
0928 08:57:57.248187 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:57.248189 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:57.248206 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:57.248281 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:57.248828 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.249683 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.250530 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.250830 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.250933 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:57.253147 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:57.253157 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:57.253157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:57.253166 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:57.254640 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:57.255048 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:57.255051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:57.255052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:57.255159 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:57.255457 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:57.255457 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:57.255458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:57.255747 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:57.255830 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:57.256030 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 08:57:57.256241 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":115,"thread_start_us":73,"threads_started":1}
I20260928 08:57:57.257372 14044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.167.253" port: 43757 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:57:57.257741 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.257626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:43757 (local address 127.0.0.1:39358)
0928 08:57:57.257670 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:57.257717 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.253:43757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 08:57:57.257867 13992 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.167.253:43757: connect: Connection refused (error 111)
I20260928 08:57:57.258266 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.258183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:43757 (local address 127.0.0.1:39362)
0928 08:57:57.258220 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:57.258245 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.253:43757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 08:57:57.258358 14044 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.167.253:43757: Network error: Client connection negotiation failed: client connection to 127.23.167.253:43757: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.23.167.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.253.keytab.
Entry for principal kudu/127.23.167.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.253.keytab.
I20260928 08:57:57.275830 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.253:43757
--webserver_interface=127.23.167.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.253.keytab
--principal=kudu/127.23.167.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:37415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:43415,127.23.167.253:43757,127.23.167.252:39927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:57:57.307585 14044 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.167.253" port: 43757 } attempt: 1
I20260928 08:57:57.308070 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.307939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:43757 (local address 127.0.0.1:39368)
0928 08:57:57.308004 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:57.308038 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.253:43757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 08:57:57.308367 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.308281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:43757 (local address 127.0.0.1:39380)
0928 08:57:57.308319 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:57.308343 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.253:43757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 08:57:57.308451 14044 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.167.253:43757: Network error: Client connection negotiation failed: client connection to 127.23.167.253:43757: connect: Connection refused (error 111)
W20260928 08:57:57.358417 14051 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 08:57:57.358662 14051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:57.358687 14051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:57.358716 14051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:57.359956 14051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:57:57.360010 14051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:57.360029 14051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:57.360046 14051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:57:57.360060 14051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:57:57.361613 14051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:37415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:43415,127.23.167.253:43757,127.23.167.252:39927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.253.keytab
--principal=kudu/127.23.167.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.253:43757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.167.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:57.361871 14051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:57.362094 14051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:57.366765 14059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:57.366930 14051 server_base.cc:1034] running on GCE node
W20260928 08:57:57.366772 14056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:57.366772 14057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:57.367246 14051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:57.367514 14051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:57.368642 14051 hybrid_clock.cc:648] HybridClock initialized: now 1790585877368628 us; error 28 us; skew 500 ppm
Sep 28 08:57:57 dist-test-slave-pbjp krb5kdc[13963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585877, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:57.370970 14051 init.cc:377] Logged in from keytab as kudu/127.23.167.253@KRBTEST.COM (short username kudu)
I20260928 08:57:57.371397 14051 webserver.cc:460] Webserver started at http://127.23.167.253:36793/ using document root <none> and password file <none>
I20260928 08:57:57.371606 14051 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:57.371650 14051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:57.371768 14051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:57.372777 14051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-1/data/instance:
uuid: "5eb9a512b45c4b9882adfdab958a7482"
format_stamp: "Formatted at 2026-09-28 08:57:57 on dist-test-slave-pbjp"
server_key: "76d335d1196389afc6f0b4cb1c1cebfc"
server_key_iv: "763c7060fee4a9891cbc9f99d61ac792"
server_key_version: "encryptionkey@0"
I20260928 08:57:57.373134 14051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-1/wal/instance:
uuid: "5eb9a512b45c4b9882adfdab958a7482"
format_stamp: "Formatted at 2026-09-28 08:57:57 on dist-test-slave-pbjp"
server_key: "76d335d1196389afc6f0b4cb1c1cebfc"
server_key_iv: "763c7060fee4a9891cbc9f99d61ac792"
server_key_version: "encryptionkey@0"
I20260928 08:57:57.374325 14051 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:57:57.375010 14066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:57.375177 14051 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:57:57.375257 14051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-1/wal
uuid: "5eb9a512b45c4b9882adfdab958a7482"
format_stamp: "Formatted at 2026-09-28 08:57:57 on dist-test-slave-pbjp"
server_key: "76d335d1196389afc6f0b4cb1c1cebfc"
server_key_iv: "763c7060fee4a9891cbc9f99d61ac792"
server_key_version: "encryptionkey@0"
I20260928 08:57:57.375308 14051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:57.376593 14044 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.167.253" port: 43757 } attempt: 2
I20260928 08:57:57.376950 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.376835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:43757 (local address 127.0.0.1:39394)
0928 08:57:57.376885 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:57.376914 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.253:43757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 08:57:57.377293 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.377221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:43757 (local address 127.0.0.1:39400)
0928 08:57:57.377257 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:57.377278 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.253:43757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 08:57:57.377380 14044 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.167.253:43757: Network error: Client connection negotiation failed: client connection to 127.23.167.253:43757: connect: Connection refused (error 111)
I20260928 08:57:57.395570 14051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:57.397387 14051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:57.397519 14051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:57.401180 14051 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.253:43757
I20260928 08:57:57.401238 14118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.253:43757 every 8 connection(s)
I20260928 08:57:57.401556 14051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-1/data/info.pb
I20260928 08:57:57.402397 14119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:57.403851 14119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43415 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:57:57 dist-test-slave-pbjp krb5kdc[13963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585877, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.253@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:57:57.410741 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 14051
I20260928 08:57:57.410832 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-1/wal/instance
I20260928 08:57:57.411038 24223 external_mini_cluster.cc:1428] Setting key 5cf91ffb3349a385ecda9ee13636c1d6
I20260928 08:57:57.412094 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.404490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41560 (local address 127.23.167.254:43415)
0928 08:57:57.404546 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:57.404548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:57.404721 (+ 173us) server_negotiation.cc:407] Connection header received
0928 08:57:57.405278 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:57.405280 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:57.405296 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:57.405344 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:57.405952 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.406427 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.407374 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.407589 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.407644 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:57.409640 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:57.409641 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:57.409642 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:57.409648 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:57.410722 (+ 1074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:57.411165 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:57.411166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:57.411166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:57.411220 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:57.411457 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:57.411458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:57.411458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:57.411593 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:57.411648 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:57.411921 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 08:57:57.412005 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 08:57:57.412279 14121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.404448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:43415 (local address 127.0.0.1:41560)
0928 08:57:57.404605 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:57.404644 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:57.405181 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:57.405428 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:57.405432 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:57.405672 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:57:57.405862 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:57.405865 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.406507 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.406508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:57.407292 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:57.407294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.407658 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.407659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:57.407783 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:57.408241 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:57:57.408249 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:57:57.409547 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:57:57.410798 (+ 1251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:57.410806 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:57.410809 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:57.411092 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:57.411278 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:57.411278 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:57.411279 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:57.411405 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:57.411657 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:57:57.411666 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:57:57.411801 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 08:57:57.411892 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 08:57:57.412060 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":71,"threads_started":1}
I20260928 08:57:57.412801 14119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.167.253" port: 43757 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:57:57 dist-test-slave-pbjp krb5kdc[13963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585877, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.253@KRBTEST.COM for kudu/127.23.167.253@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.167.252@KRBTEST.COM; defaulting to no policy
I20260928 08:57:57.420517 14121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.413032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:43757 (local address 127.0.0.1:39404)
0928 08:57:57.413092 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:57.413103 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:57.413139 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:57.413436 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:57.413437 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:57.413567 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:57:57.413621 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:57.413623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.414760 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.414760 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:57.415286 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:57.415288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.415974 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.415975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:57.416032 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:57.416451 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:57:57.416455 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:57:57.417785 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:57:57.419443 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:57.419444 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:57.419445 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:57.419601 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:57.419791 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:57.419792 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:57.419792 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:57.419862 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:57.420191 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:57:57.420192 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:57:57.420232 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 08:57:57.420293 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 08:57:57.420398 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 08:57:57.420545 14123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.413159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39404 (local address 127.23.167.253:43757)
0928 08:57:57.413297 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:57.413302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:57.413317 (+ 15us) server_negotiation.cc:407] Connection header received
0928 08:57:57.413335 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:57.413337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:57.413347 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:57.413420 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:57.413683 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.414679 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.415407 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.415896 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.415974 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:57.417861 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:57.417869 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:57.417869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:57.417875 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:57.419323 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:57.419668 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:57.419671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:57.419671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:57.419719 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:57.419932 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:57.419933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:57.419933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:57.420125 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:57.420204 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:57.420332 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 08:57:57.420451 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
I20260928 08:57:57.421185 14119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.167.252" port: 39927 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.252@KRBTEST.COM" created.
I20260928 08:57:57.421542 14121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.421392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:39927 (local address 127.0.0.1:34356)
0928 08:57:57.421438 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:57.421515 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.252:39927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 08:57:57.421674 14070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.167.252:39927: connect: Connection refused (error 111)
I20260928 08:57:57.422092 14121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.421987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:39927 (local address 127.0.0.1:34364)
0928 08:57:57.422038 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:57.422068 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.252:39927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 08:57:57.422188 14119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.167.252:39927: Network error: Client connection negotiation failed: client connection to 127.23.167.252:39927: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.252.keytab.
Entry for principal kudu/127.23.167.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.252.keytab.
I20260928 08:57:57.435457 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.252:39927
--webserver_interface=127.23.167.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.252.keytab
--principal=kudu/127.23.167.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:37415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:43415,127.23.167.253:43757,127.23.167.252:39927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:57:57.468554 14044 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.167.253" port: 43757 } attempt: 3
I20260928 08:57:57.471391 14119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.167.252" port: 39927 } attempt: 1
I20260928 08:57:57.471771 14121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.471664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:39927 (local address 127.0.0.1:34370)
0928 08:57:57.471712 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:57.471742 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.252:39927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 08:57:57.472105 14121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.472025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:39927 (local address 127.0.0.1:34374)
0928 08:57:57.472062 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:57.472085 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.252:39927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 08:57:57.472194 14119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.167.252:39927: Network error: Client connection negotiation failed: client connection to 127.23.167.252:39927: connect: Connection refused (error 111)
Sep 28 08:57:57 dist-test-slave-pbjp krb5kdc[13963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585877, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for kudu/127.23.167.253@KRBTEST.COM
I20260928 08:57:57.475983 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.468911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:43757 (local address 127.0.0.1:39420)
0928 08:57:57.468969 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:57.468982 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:57.469036 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:57.469301 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:57.469303 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:57.469503 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:57:57.469578 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:57.469580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.470531 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.470532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:57.471065 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:57.471067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.471645 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.471647 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:57.471741 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:57.472324 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:57:57.472329 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:57:57.473650 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:57:57.475174 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:57.475175 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:57.475175 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:57.475327 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:57.475501 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:57.475502 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:57.475502 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:57.475547 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:57.475763 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:57:57.475764 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:57:57.475800 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 08:57:57.475842 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 08:57:57.475913 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 08:57:57.476018 14123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.469009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39420 (local address 127.23.167.253:43757)
0928 08:57:57.469085 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:57.469087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:57.469099 (+ 12us) server_negotiation.cc:407] Connection header received
0928 08:57:57.469130 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:57.469132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:57.469143 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:57.469288 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:57.469654 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.470451 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.471145 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.471496 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.471571 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:57.473738 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:57.473739 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:57.473740 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:57.473747 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:57.475105 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:57.475392 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:57.475393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:57.475394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:57.475448 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:57.475591 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:57.475592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:57.475592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:57.475712 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:57.475765 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:57.475866 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 08:57:57.475934 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 08:57:57.476425 14044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.167.252" port: 39927 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:57:57.476722 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.476645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:39927 (local address 127.0.0.1:34382)
0928 08:57:57.476675 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:57.476702 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.252:39927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260928 08:57:57.477010 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.476930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:39927 (local address 127.0.0.1:34398)
0928 08:57:57.476962 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:57.476993 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.252:39927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 08:57:57.477102 14044 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.167.252:39927: Network error: Client connection negotiation failed: client connection to 127.23.167.252:39927: connect: Connection refused (error 111)
I20260928 08:57:57.508242 14044 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.167.252" port: 39927 } attempt: 1
I20260928 08:57:57.508672 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.508555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:39927 (local address 127.0.0.1:34400)
0928 08:57:57.508611 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:57.508641 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.252:39927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 08:57:57.509007 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.508925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:39927 (local address 127.0.0.1:34408)
0928 08:57:57.508964 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:57.508988 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.252:39927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 08:57:57.509096 14044 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.167.252:39927: Network error: Client connection negotiation failed: client connection to 127.23.167.252:39927: connect: Connection refused (error 111)
W20260928 08:57:57.517614 14126 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 08:57:57.517844 14126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:57.517866 14126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:57.517903 14126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:57.519098 14126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:57:57.519140 14126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:57.519160 14126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:57.519176 14126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:57:57.519191 14126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:57:57.520709 14126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:37415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:43415,127.23.167.253:43757,127.23.167.252:39927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.252.keytab
--principal=kudu/127.23.167.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.252:39927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.167.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:57.520961 14126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:57.521205 14126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:57:57.525693 14132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:57.525800 14134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:57.525701 14131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:57.525954 14126 server_base.cc:1034] running on GCE node
I20260928 08:57:57.526180 14126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:57.526471 14126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:57.527609 14126 hybrid_clock.cc:648] HybridClock initialized: now 1790585877527597 us; error 26 us; skew 500 ppm
Sep 28 08:57:57 dist-test-slave-pbjp krb5kdc[13963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585877, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:57.530077 14126 init.cc:377] Logged in from keytab as kudu/127.23.167.252@KRBTEST.COM (short username kudu)
I20260928 08:57:57.530519 14126 webserver.cc:460] Webserver started at http://127.23.167.252:34137/ using document root <none> and password file <none>
I20260928 08:57:57.530748 14126 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:57.530797 14126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:57.530928 14126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:57.531949 14126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-2/data/instance:
uuid: "2fa254d5ce96410d93ac456f67c7ba9d"
format_stamp: "Formatted at 2026-09-28 08:57:57 on dist-test-slave-pbjp"
server_key: "5674f602651b2aea20a268461d5c5b85"
server_key_iv: "0fe7d6c03e3f6c8f63bf6f5a2c76b9ac"
server_key_version: "encryptionkey@0"
I20260928 08:57:57.532267 14126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-2/wal/instance:
uuid: "2fa254d5ce96410d93ac456f67c7ba9d"
format_stamp: "Formatted at 2026-09-28 08:57:57 on dist-test-slave-pbjp"
server_key: "5674f602651b2aea20a268461d5c5b85"
server_key_iv: "0fe7d6c03e3f6c8f63bf6f5a2c76b9ac"
server_key_version: "encryptionkey@0"
I20260928 08:57:57.533532 14126 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:57:57.534164 14141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:57.534303 14126 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:57:57.534382 14126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-2/wal
uuid: "2fa254d5ce96410d93ac456f67c7ba9d"
format_stamp: "Formatted at 2026-09-28 08:57:57 on dist-test-slave-pbjp"
server_key: "5674f602651b2aea20a268461d5c5b85"
server_key_iv: "0fe7d6c03e3f6c8f63bf6f5a2c76b9ac"
server_key_version: "encryptionkey@0"
I20260928 08:57:57.534443 14126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:57.540315 14119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.167.252" port: 39927 } attempt: 2
I20260928 08:57:57.540665 14121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.540563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:39927 (local address 127.0.0.1:34412)
0928 08:57:57.540610 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:57.540639 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.252:39927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 08:57:57.540948 14121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.540871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:39927 (local address 127.0.0.1:34418)
0928 08:57:57.540904 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:57.540927 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.252:39927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260928 08:57:57.541028 14119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.167.252:39927: Network error: Client connection negotiation failed: client connection to 127.23.167.252:39927: connect: Connection refused (error 111)
I20260928 08:57:57.556069 14126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:57.558238 14126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:57.558383 14126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:57.562180 14126 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.252:39927
I20260928 08:57:57.562212 14193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.252:39927 every 8 connection(s)
I20260928 08:57:57.562539 14126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-2/data/info.pb
I20260928 08:57:57.563421 14194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:57.564963 14194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43415 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:57:57.570227 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 14126
I20260928 08:57:57.570322 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585589999956-24223-0/minicluster-data/master-2/wal/instance
Sep 28 08:57:57 dist-test-slave-pbjp krb5kdc[13963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585877, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.252@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:57:57.570519 24223 external_mini_cluster.cc:1428] Setting key 7c5edc284f3100c00a88426c377671af
I20260928 08:57:57.573640 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.565678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41570 (local address 127.23.167.254:43415)
0928 08:57:57.565729 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:57.565730 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:57.565929 (+ 199us) server_negotiation.cc:407] Connection header received
0928 08:57:57.566402 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:57.566403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:57.566414 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:57.566472 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:57.567082 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.567548 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.568494 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.568677 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.568728 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:57.570925 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:57.570927 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:57.570927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:57.570935 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:57.572316 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:57.572734 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:57.572735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:57.572736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:57.572797 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:57.573015 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:57.573015 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:57.573016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:57.573174 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:57.573239 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:57.573458 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 08:57:57.573548 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 08:57:57.573781 14196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.565649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:43415 (local address 127.0.0.1:41570)
0928 08:57:57.565812 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:57.565854 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:57.566324 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:57.566550 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:57.566554 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:57.566792 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:57:57.566988 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:57.566993 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.567630 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.567630 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:57.568421 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:57.568423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.568735 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.568736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:57.568835 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:57.569352 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:57:57.569361 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:57:57.570807 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:57:57.572390 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:57.572396 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:57.572398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:57.572674 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:57.572855 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:57.572856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:57.572856 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:57.572951 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:57.573229 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:57:57.573232 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:57:57.573345 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 08:57:57.573426 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 08:57:57.573559 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260928 08:57:57.574285 14194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.167.253" port: 43757 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:57:57 dist-test-slave-pbjp krb5kdc[13963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.252@KRBTEST.COM
Sep 28 08:57:57 dist-test-slave-pbjp krb5kdc[13963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585877, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.252@KRBTEST.COM for kudu/127.23.167.253@KRBTEST.COM
I20260928 08:57:57.579407 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.570879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34420 (local address 127.23.167.252:39927)
0928 08:57:57.571032 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:57.571037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:57.571057 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:57:57.571122 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:57.571123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:57.571134 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:57.571193 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:57.571805 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.572594 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.573532 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.574089 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.574152 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:57.576745 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:57.576753 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:57.576754 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:57.576760 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:57.578225 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:57.578599 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:57.578601 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:57.578602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:57.578652 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:57.578840 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:57.578841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:57.578842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:57.579004 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:57.579145 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:57.579227 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 08:57:57.579315 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":61,"threads_started":1}
I20260928 08:57:57.580801 14123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.574587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39426 (local address 127.23.167.253:43757)
0928 08:57:57.574642 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:57.574644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:57.574651 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:57:57.574676 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:57.574677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:57.574686 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:57.574727 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:57.574949 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.575655 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.576547 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.576754 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.576807 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:57.578436 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:57.578437 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:57.578437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:57.578445 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:57.579783 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:57.580128 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:57.580129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:57.580129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:57.580210 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:57.580367 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:57.580368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:57.580368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:57.580497 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:57.580545 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:57.580687 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 08:57:57.580739 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 08:57:57.580833 14196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.574520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:43757 (local address 127.0.0.1:39426)
0928 08:57:57.574560 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:57.574569 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:57.574599 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:57.574730 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:57.574730 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:57.574835 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:57:57.574877 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:57.574878 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.575726 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.575727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:57.576471 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:57.576474 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.576812 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.576813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:57.576878 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:57.577323 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:57:57.577327 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:57:57.578363 (+ 1036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:57:57.579848 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:57.579849 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:57.579849 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:57.580062 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:57.580270 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:57.580270 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:57.580271 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:57.580326 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:57.580553 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:57:57.580554 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:57:57.580596 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 08:57:57.580671 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 08:57:57.580750 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 08:57:57.581146 14194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.167.252" port: 39927 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:57:57 dist-test-slave-pbjp krb5kdc[13963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585877, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.252@KRBTEST.COM for kudu/127.23.167.252@KRBTEST.COM
I20260928 08:57:57.584231 14044 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.167.252" port: 39927 } attempt: 2
Sep 28 08:57:57 dist-test-slave-pbjp krb5kdc[13963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:57:57.587563 14196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.581353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:39927 (local address 127.0.0.1:34430)
0928 08:57:57.581391 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:57.581401 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:57.581434 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:57.581531 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:57.581531 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:57.581665 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:57:57.581694 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:57.581695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.582237 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.582238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:57.582688 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:57.582689 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.583003 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.583004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:57.583049 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:57.583387 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:57:57.583392 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:57:57.585036 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:57:57.586659 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:57.586660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:57.586661 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:57.586840 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:57.586994 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:57.586994 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:57.586995 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:57.587044 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:57.587284 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:57:57.587285 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:57:57.587325 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 08:57:57.587387 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 08:57:57.587476 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 08:57:57.587790 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.581396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34430 (local address 127.23.167.252:39927)
0928 08:57:57.581426 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:57.581427 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:57.581434 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:57:57.581477 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:57.581477 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:57.581490 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:57.581527 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:57.581753 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.582176 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.582750 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.582946 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.582995 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:57.585136 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:57.585137 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:57.585138 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:57.585145 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:57.586566 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:57.586905 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:57.586906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:57.586906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:57.586951 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:57.587100 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:57.587101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:57.587101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:57.587219 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:57.587280 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:57.587636 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 08:57:57.587708 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 08:57:57.587998 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.580288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41586 (local address 127.23.167.254:43415)
0928 08:57:57.580333 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:57.580334 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:57.580341 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:57:57.580362 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:57.580362 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:57.580371 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:57.580420 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:57.581405 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.582000 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.582630 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.582822 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.582873 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:57.585268 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:57.585269 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:57.585270 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:57.585276 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:57.586560 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:57.586925 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:57.586926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:57.586926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:57.586991 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:57.587142 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:57.587143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:57.587143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:57.587253 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:57.587424 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:57.587809 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 08:57:57.587904 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
Sep 28 08:57:57 dist-test-slave-pbjp krb5kdc[13963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585877, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for kudu/127.23.167.252@KRBTEST.COM
I20260928 08:57:57.589604 14194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fa254d5ce96410d93ac456f67c7ba9d: Bootstrap starting.
I20260928 08:57:57.590260 14194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fa254d5ce96410d93ac456f67c7ba9d: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:57.590525 14194 log.cc:824] T 00000000000000000000000000000000 P 2fa254d5ce96410d93ac456f67c7ba9d: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:57.591301 14194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fa254d5ce96410d93ac456f67c7ba9d: No bootstrap required, opened a new log
I20260928 08:57:57.591643 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.584451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:39927 (local address 127.0.0.1:34436)
0928 08:57:57.584496 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:57.584506 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:57.584536 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:57.584729 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:57.584730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:57.584876 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:57:57.584914 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:57.584915 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.585742 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.585744 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:57.586825 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:57.586828 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.587246 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.587248 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:57.587302 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:57.587633 (+ 331us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:57:57.587637 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:57:57.588936 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:57:57.590646 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:57.590647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:57.590647 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:57.590837 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:57.591034 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:57.591035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:57.591035 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:57.591093 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:57.591347 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:57:57.591348 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:57:57.591392 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 08:57:57.591459 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 08:57:57.591555 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 08:57:57.591681 14199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.584499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34436 (local address 127.23.167.252:39927)
0928 08:57:57.584636 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:57.584638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:57.584642 (+ 4us) server_negotiation.cc:407] Connection header received
0928 08:57:57.584656 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:57.584656 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:57.584663 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:57.584715 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:57.584996 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.585671 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.586887 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.587166 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.587426 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:57.589026 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:57.589027 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:57.589028 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:57.589034 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:57.590530 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:57.590904 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:57.590905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:57.590906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:57.590963 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:57.591153 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:57.591154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:57.591154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:57.591288 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:57.591353 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:57.591487 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 08:57:57.591555 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":98,"threads_started":1}
I20260928 08:57:57.593103 14044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bf5556a2e944ac99bf1c7cff58be072: Bootstrap starting.
Sep 28 08:57:57 dist-test-slave-pbjp krb5kdc[13963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.253@KRBTEST.COM
I20260928 08:57:57.593200 14194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fa254d5ce96410d93ac456f67c7ba9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf5556a2e944ac99bf1c7cff58be072" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43415 } } peers { permanent_uuid: "5eb9a512b45c4b9882adfdab958a7482" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 43757 } } peers { permanent_uuid: "2fa254d5ce96410d93ac456f67c7ba9d" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 39927 } }
I20260928 08:57:57.593358 14194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fa254d5ce96410d93ac456f67c7ba9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fa254d5ce96410d93ac456f67c7ba9d, State: Initialized, Role: FOLLOWER
I20260928 08:57:57.593432 14194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fa254d5ce96410d93ac456f67c7ba9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf5556a2e944ac99bf1c7cff58be072" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43415 } } peers { permanent_uuid: "5eb9a512b45c4b9882adfdab958a7482" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 43757 } } peers { permanent_uuid: "2fa254d5ce96410d93ac456f67c7ba9d" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 39927 } }
I20260928 08:57:57.593706 14194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fa254d5ce96410d93ac456f67c7ba9d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:57:57.593659 14201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fa254d5ce96410d93ac456f67c7ba9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf5556a2e944ac99bf1c7cff58be072" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43415 } } peers { permanent_uuid: "5eb9a512b45c4b9882adfdab958a7482" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 43757 } } peers { permanent_uuid: "2fa254d5ce96410d93ac456f67c7ba9d" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 39927 } } }
I20260928 08:57:57.593818 14044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bf5556a2e944ac99bf1c7cff58be072: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:57.594123 14044 log.cc:824] T 00000000000000000000000000000000 P 0bf5556a2e944ac99bf1c7cff58be072: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:57.594137 14201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fa254d5ce96410d93ac456f67c7ba9d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:57:57.594913 14044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bf5556a2e944ac99bf1c7cff58be072: No bootstrap required, opened a new log
W20260928 08:57:57.595897 14213 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2fa254d5ce96410d93ac456f67c7ba9d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:57:57.595991 14213 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:57:57.596844 14123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.588796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39442 (local address 127.23.167.253:43757)
0928 08:57:57.588828 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:57.588829 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:57.588838 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:57:57.588879 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:57.588880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:57.588888 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:57.588939 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:57.589501 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.590166 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.591004 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.591206 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.591255 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:57.593681 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:57.593682 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:57.593683 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:57.593689 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:57.595133 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:57.595508 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:57.595509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:57.595510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:57.595571 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:57.595781 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:57.595782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:57.595782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:57.595909 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:57.596075 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:57.596682 (+ 607us) server_negotiation.cc:299] Negotiation successful
0928 08:57:57.596757 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 08:57:57.596887 14044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bf5556a2e944ac99bf1c7cff58be072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf5556a2e944ac99bf1c7cff58be072" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43415 } } peers { permanent_uuid: "5eb9a512b45c4b9882adfdab958a7482" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 43757 } } peers { permanent_uuid: "2fa254d5ce96410d93ac456f67c7ba9d" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 39927 } }
I20260928 08:57:57.597008 14044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bf5556a2e944ac99bf1c7cff58be072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bf5556a2e944ac99bf1c7cff58be072, State: Initialized, Role: FOLLOWER
I20260928 08:57:57.597113 14044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bf5556a2e944ac99bf1c7cff58be072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf5556a2e944ac99bf1c7cff58be072" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43415 } } peers { permanent_uuid: "5eb9a512b45c4b9882adfdab958a7482" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 43757 } } peers { permanent_uuid: "2fa254d5ce96410d93ac456f67c7ba9d" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 39927 } }
I20260928 08:57:57.597410 14044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bf5556a2e944ac99bf1c7cff58be072 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:57:57.596437 14126 master_runner.cc:186] Error getting master registration for 127.23.167.254:43415: OK, instance_id { permanent_uuid: "0bf5556a2e944ac99bf1c7cff58be072" instance_seqno: 1790585877242802 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 08:57:57.598654 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 08:57:57.599009 14226 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0bf5556a2e944ac99bf1c7cff58be072: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:57:57.599066 14226 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:57:57.599248 24223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:57:57.600265 14018 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0bf5556a2e944ac99bf1c7cff58be072"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41586
I20260928 08:57:57.600379 14018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bf5556a2e944ac99bf1c7cff58be072 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 08:57:57.600425 14018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bf5556a2e944ac99bf1c7cff58be072 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:57.601604 14018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bf5556a2e944ac99bf1c7cff58be072 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf5556a2e944ac99bf1c7cff58be072" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43415 } } peers { permanent_uuid: "5eb9a512b45c4b9882adfdab958a7482" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 43757 } } peers { permanent_uuid: "2fa254d5ce96410d93ac456f67c7ba9d" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 39927 } }
I20260928 08:57:57.601851 14018 leader_election.cc:288] T 00000000000000000000000000000000 P 0bf5556a2e944ac99bf1c7cff58be072 [CANDIDATE]: Term 1 election: Requested vote from peers 5eb9a512b45c4b9882adfdab958a7482 (127.23.167.253:43757), 2fa254d5ce96410d93ac456f67c7ba9d (127.23.167.252:39927)
I20260928 08:57:57.602176 14168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bf5556a2e944ac99bf1c7cff58be072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2fa254d5ce96410d93ac456f67c7ba9d"
I20260928 08:57:57.602293 14168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fa254d5ce96410d93ac456f67c7ba9d [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:57.603293 14168 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2fa254d5ce96410d93ac456f67c7ba9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bf5556a2e944ac99bf1c7cff58be072 in term 1.
I20260928 08:57:57.603670 13995 leader_election.cc:302] T 00000000000000000000000000000000 P 0bf5556a2e944ac99bf1c7cff58be072 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bf5556a2e944ac99bf1c7cff58be072, 2fa254d5ce96410d93ac456f67c7ba9d; no voters:
I20260928 08:57:57.604192 14228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bf5556a2e944ac99bf1c7cff58be072 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:57:57.604427 14228 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bf5556a2e944ac99bf1c7cff58be072 [term 1 LEADER]: Becoming Leader. State: Replica: 0bf5556a2e944ac99bf1c7cff58be072, State: Running, Role: LEADER
I20260928 08:57:57.604575 14228 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bf5556a2e944ac99bf1c7cff58be072 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf5556a2e944ac99bf1c7cff58be072" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43415 } } peers { permanent_uuid: "5eb9a512b45c4b9882adfdab958a7482" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 43757 } } peers { permanent_uuid: "2fa254d5ce96410d93ac456f67c7ba9d" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 39927 } }
I20260928 08:57:57.605211 14228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bf5556a2e944ac99bf1c7cff58be072 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bf5556a2e944ac99bf1c7cff58be072. Latest consensus state: current_term: 1 leader_uuid: "0bf5556a2e944ac99bf1c7cff58be072" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf5556a2e944ac99bf1c7cff58be072" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43415 } } peers { permanent_uuid: "5eb9a512b45c4b9882adfdab958a7482" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 43757 } } peers { permanent_uuid: "2fa254d5ce96410d93ac456f67c7ba9d" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 39927 } } }
I20260928 08:57:57.605352 14228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bf5556a2e944ac99bf1c7cff58be072 [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:57.605880 14215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bf5556a2e944ac99bf1c7cff58be072 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bf5556a2e944ac99bf1c7cff58be072" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf5556a2e944ac99bf1c7cff58be072" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43415 } } peers { permanent_uuid: "5eb9a512b45c4b9882adfdab958a7482" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 43757 } } peers { permanent_uuid: "2fa254d5ce96410d93ac456f67c7ba9d" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 39927 } } }
I20260928 08:57:57.605975 14215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bf5556a2e944ac99bf1c7cff58be072 [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:57.606424 13976 master_runner.cc:186] Error getting master registration for 127.23.167.253:43757: OK, instance_id { permanent_uuid: "5eb9a512b45c4b9882adfdab958a7482" instance_seqno: 1790585877400105 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 08:57:57.608748 13976 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 08:57:57.610203 14235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:57:57.610438 14235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:57:57.617493 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.608841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41602 (local address 127.23.167.254:43415)
0928 08:57:57.608893 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:57.608895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:57.609334 (+ 439us) server_negotiation.cc:407] Connection header received
0928 08:57:57.609364 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:57.609365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:57.609375 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:57.609432 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:57.610705 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.611474 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.612526 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.612821 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.612856 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:57.614617 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:57.614618 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:57.614619 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:57.614626 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:57.616286 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:57.616668 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:57.616669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:57.616670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:57.616740 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:57.616912 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:57.616913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:57.616913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:57.617100 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:57.617215 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:57.617326 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 08:57:57.617398 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 08:57:57.620698 14199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.609286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34450 (local address 127.23.167.252:39927)
0928 08:57:57.609344 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:57.609345 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:57.610023 (+ 678us) server_negotiation.cc:407] Connection header received
0928 08:57:57.610090 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:57.610091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:57.610103 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:57.610152 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:57.610855 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.611421 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.612389 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.612696 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.612754 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:57.617937 (+ 5183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:57.617939 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:57.617939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:57.617946 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:57.619481 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:57.619853 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:57.619855 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:57.619855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:57.619924 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:57.620112 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:57.620112 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:57.620113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:57.620242 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:57.620313 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:57.620515 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 08:57:57.620601 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 08:57:57.624686 14123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.609787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39446 (local address 127.23.167.253:43757)
0928 08:57:57.609845 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:57.609847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:57.617464 (+ 7617us) server_negotiation.cc:407] Connection header received
0928 08:57:57.617818 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:57.617820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:57.617832 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:57.617860 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:57.618377 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.619112 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.620106 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.620414 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.620478 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:57.622076 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:57.622077 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:57.622077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:57.622085 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:57.623593 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:57.623944 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:57.623945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:57.623945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:57.623998 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:57.624166 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:57.624167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:57.624167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:57.624298 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:57.624391 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:57.624505 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 08:57:57.624591 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 08:57:57.632194 14119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.167.252" port: 39927 } attempt: 3
I20260928 08:57:57.633591 14168 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2fa254d5ce96410d93ac456f67c7ba9d [term 1 FOLLOWER]: Refusing update from remote peer 0bf5556a2e944ac99bf1c7cff58be072: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:57:57.634367 14229 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0bf5556a2e944ac99bf1c7cff58be072 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fa254d5ce96410d93ac456f67c7ba9d" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 39927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:57:57.635879 14201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fa254d5ce96410d93ac456f67c7ba9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bf5556a2e944ac99bf1c7cff58be072. Latest consensus state: current_term: 1 leader_uuid: "0bf5556a2e944ac99bf1c7cff58be072" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf5556a2e944ac99bf1c7cff58be072" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43415 } } peers { permanent_uuid: "5eb9a512b45c4b9882adfdab958a7482" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 43757 } } peers { permanent_uuid: "2fa254d5ce96410d93ac456f67c7ba9d" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 39927 } } }
I20260928 08:57:57.635967 14201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fa254d5ce96410d93ac456f67c7ba9d [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 08:57:57 dist-test-slave-pbjp krb5kdc[13963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585877, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.253@KRBTEST.COM for kudu/127.23.167.252@KRBTEST.COM
I20260928 08:57:57.638351 14215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bf5556a2e944ac99bf1c7cff58be072 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0bf5556a2e944ac99bf1c7cff58be072" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf5556a2e944ac99bf1c7cff58be072" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43415 } } peers { permanent_uuid: "5eb9a512b45c4b9882adfdab958a7482" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 43757 } } peers { permanent_uuid: "2fa254d5ce96410d93ac456f67c7ba9d" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 39927 } } }
I20260928 08:57:57.638453 14215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bf5556a2e944ac99bf1c7cff58be072 [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:57.638923 14201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fa254d5ce96410d93ac456f67c7ba9d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0bf5556a2e944ac99bf1c7cff58be072" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf5556a2e944ac99bf1c7cff58be072" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43415 } } peers { permanent_uuid: "5eb9a512b45c4b9882adfdab958a7482" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 43757 } } peers { permanent_uuid: "2fa254d5ce96410d93ac456f67c7ba9d" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 39927 } } }
I20260928 08:57:57.639014 14201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fa254d5ce96410d93ac456f67c7ba9d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:57:57.639313 14239 mvcc.cc:205] Tried to move back new op lower bound from 7334239754783399936 to 7334239754669617152. Current Snapshot: MvccSnapshot[applied={T|T < 7334239754783399936}]
I20260928 08:57:57.639454 14240 mvcc.cc:205] Tried to move back new op lower bound from 7334239754783399936 to 7334239754669617152. Current Snapshot: MvccSnapshot[applied={T|T < 7334239754783399936}]
I20260928 08:57:57.641119 14235 catalog_manager.cc:1347] Generated new cluster ID: b385276130e041f1bac178c70f6c62a2
I20260928 08:57:57.641193 14235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:57:57.641330 14121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.632465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:39927 (local address 127.0.0.1:34458)
0928 08:57:57.632549 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:57.632560 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:57.632593 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:57.632809 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:57.632810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:57.632965 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:57:57.633010 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:57.633011 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.634094 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.634095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:57.634910 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:57.634912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.635535 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.635536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:57.635592 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:57.636109 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:57:57.636115 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:57:57.637845 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:57:57.640282 (+ 2437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:57.640283 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:57.640283 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:57.640502 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:57.640691 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:57.640692 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:57.640692 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:57.640744 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:57.641000 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:57:57.641001 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:57:57.641043 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 08:57:57.641121 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 08:57:57.641218 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 08:57:57.641878 14199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:57.632660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34458 (local address 127.23.167.252:39927)
0928 08:57:57.632694 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:57.632695 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:57.632701 (+ 6us) server_negotiation.cc:407] Connection header received
0928 08:57:57.632720 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:57.632721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:57.632729 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:57.632771 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:57.633126 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.633912 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.635116 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:57.635425 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:57.635482 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:57.637933 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:57.637935 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:57.637935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:57.637941 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:57.639596 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:57.640579 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:57.640581 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:57.640581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:57.640633 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:57.640801 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:57.640801 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:57.640802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:57.640922 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:57.640974 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:57.641719 (+ 745us) server_negotiation.cc:299] Negotiation successful
0928 08:57:57.641794 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 08:57:57.643282 14119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eb9a512b45c4b9882adfdab958a7482: Bootstrap starting.
I20260928 08:57:57.644055 14119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5eb9a512b45c4b9882adfdab958a7482: Neither blocks nor log segments found. Creating new log.
I20260928 08:57:57.644362 14119 log.cc:824] T 00000000000000000000000000000000 P 5eb9a512b45c4b9882adfdab958a7482: Log is configured to *not* fsync() on all Append() calls
I20260928 08:57:57.645412 14119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eb9a512b45c4b9882adfdab958a7482: No bootstrap required, opened a new log
I20260928 08:57:57.647392 14119 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5eb9a512b45c4b9882adfdab958a7482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf5556a2e944ac99bf1c7cff58be072" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43415 } } peers { permanent_uuid: "5eb9a512b45c4b9882adfdab958a7482" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 43757 } } peers { permanent_uuid: "2fa254d5ce96410d93ac456f67c7ba9d" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 39927 } }
I20260928 08:57:57.647519 14119 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5eb9a512b45c4b9882adfdab958a7482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eb9a512b45c4b9882adfdab958a7482, State: Initialized, Role: FOLLOWER
I20260928 08:57:57.647588 14119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5eb9a512b45c4b9882adfdab958a7482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf5556a2e944ac99bf1c7cff58be072" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43415 } } peers { permanent_uuid: "5eb9a512b45c4b9882adfdab958a7482" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 43757 } } peers { permanent_uuid: "2fa254d5ce96410d93ac456f67c7ba9d" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 39927 } }
I20260928 08:57:57.647877 14119 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5eb9a512b45c4b9882adfdab958a7482 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:57:57.648028 14092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5eb9a512b45c4b9882adfdab958a7482 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:57:57.648897 14092 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5eb9a512b45c4b9882adfdab958a7482 [term 1 FOLLOWER]: Refusing update from remote peer 0bf5556a2e944ac99bf1c7cff58be072: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:57:57.649113 14093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bf5556a2e944ac99bf1c7cff58be072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5eb9a512b45c4b9882adfdab958a7482"
I20260928 08:57:57.649780 14229 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0bf5556a2e944ac99bf1c7cff58be072 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eb9a512b45c4b9882adfdab958a7482" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 43757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:57:57.649860 14093 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5eb9a512b45c4b9882adfdab958a7482 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bf5556a2e944ac99bf1c7cff58be072 in term 1.
W20260928 08:57:57.650007 14072 tablet.cc:2369] T 00000000000000000000000000000000 P 5eb9a512b45c4b9882adfdab958a7482: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:57:57.652694 14245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eb9a512b45c4b9882adfdab958a7482 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bf5556a2e944ac99bf1c7cff58be072" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf5556a2e944ac99bf1c7cff58be072" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43415 } } peers { permanent_uuid: "5eb9a512b45c4b9882adfdab958a7482" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 43757 } } peers { permanent_uuid: "2fa254d5ce96410d93ac456f67c7ba9d" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 39927 } } }
I20260928 08:57:57.652895 14245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eb9a512b45c4b9882adfdab958a7482 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:57:57.652998 14245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eb9a512b45c4b9882adfdab958a7482 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bf5556a2e944ac99bf1c7cff58be072. Latest consensus state: current_term: 1 leader_uuid: "0bf5556a2e944ac99bf1c7cff58be072" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf5556a2e944ac99bf1c7cff58be072" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43415 } } peers { permanent_uuid: "5eb9a512b45c4b9882adfdab958a7482" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 43757 } } peers { permanent_uuid: "2fa254d5ce96410d93ac456f67c7ba9d" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 39927 } } }
I20260928 08:57:57.653120 14245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eb9a512b45c4b9882adfdab958a7482 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:57:57.653198 14245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eb9a512b45c4b9882adfdab958a7482 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0bf5556a2e944ac99bf1c7cff58be072" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf5556a2e944ac99bf1c7cff58be072" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43415 } } peers { permanent_uuid: "5eb9a512b45c4b9882adfdab958a7482" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 43757 } } peers { permanent_uuid: "2fa254d5ce96410d93ac456f67c7ba9d" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 39927 } } }
I20260928 08:57:57.653285 14245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eb9a512b45c4b9882adfdab958a7482 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 08:57:57.657320 14260 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5eb9a512b45c4b9882adfdab958a7482: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:57:57.657379 14260 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:57:57.658141 14235 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:57:57.658841 14235 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:57:57.663100 14257 mvcc.cc:205] Tried to move back new op lower bound from 7334239754783399936 to 7334239754669617152. Current Snapshot: MvccSnapshot[applied={T|T < 7334239754783399936}]
I20260928 08:57:57.663503 14228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bf5556a2e944ac99bf1c7cff58be072 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0bf5556a2e944ac99bf1c7cff58be072" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf5556a2e944ac99bf1c7cff58be072" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 43415 } } peers { permanent_uuid: "5eb9a512b45c4b9882adfdab958a7482" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 43757 } } peers { permanent_uuid: "2fa254d5ce96410d93ac456f67c7ba9d" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 39927 } } }
I20260928 08:57:57.663594 14228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bf5556a2e944ac99bf1c7cff58be072 [sys.catalog]: This master's current role is: LEADER
I20260928 08:57:57.675886 14235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bf5556a2e944ac99bf1c7cff58be072: Generated new TSK 0
I20260928 08:57:57.676111 14235 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 08:57:57.700398 14047 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:57:57.699021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41610 (local address 127.23.167.254:43415)
0928 08:57:57.699068 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:57.699071 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:57.699720 (+ 649us) server_negotiation.cc:407] Connection header received
0928 08:57:57.699823 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:57.699825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:57.699836 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:57.699865 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:57.700323 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41610: BlockingRecv error: recv got EOF from 127.0.0.1:41610 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 08:57:57.701238 14199 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:57:57.699403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34468 (local address 127.23.167.252:39927)
0928 08:57:57.699447 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:57.699449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:57.700615 (+ 1166us) server_negotiation.cc:407] Connection header received
0928 08:57:57.700784 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:57.700785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:57.700795 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:57.700823 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:57.701180 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34468: BlockingRecv error: recv got EOF from 127.0.0.1:34468 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 08:57:57.702208 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:57:57.701581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39460 (local address 127.23.167.253:43757)
0928 08:57:57.701620 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:57.701622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:57.701736 (+ 114us) server_negotiation.cc:407] Connection header received
0928 08:57:57.701819 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:57.701820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:57.701829 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:57.701856 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:57.702148 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39460: BlockingRecv error: recv got EOF from 127.0.0.1:39460 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 08:57:57.704427 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:57:57.703754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39470 (local address 127.23.167.253:43757)
0928 08:57:57.703790 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:57.703792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:57.703948 (+ 156us) server_negotiation.cc:407] Connection header received
0928 08:57:57.704048 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:57.704048 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:57.704067 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:57.704095 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:57.704385 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39470: BlockingRecv error: recv got EOF from 127.0.0.1:39470 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 08:57:57.707360 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:57:57.706666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39482 (local address 127.23.167.253:43757)
0928 08:57:57.706698 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:57.706700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:57.706899 (+ 199us) server_negotiation.cc:407] Connection header received
0928 08:57:57.706986 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:57.706987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:57.706997 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:57.707025 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:57.707312 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39482: BlockingRecv error: recv got EOF from 127.0.0.1:39482 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 08:57:57.711169 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:57:57.710465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39486 (local address 127.23.167.253:43757)
0928 08:57:57.710510 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:57.710511 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:57.710705 (+ 194us) server_negotiation.cc:407] Connection header received
0928 08:57:57.710792 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:57.710793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:57.710802 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:57.710828 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:57.711123 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39486: BlockingRecv error: recv got EOF from 127.0.0.1:39486 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260928 08:57:57.717322 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:57:57.716432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39496 (local address 127.23.167.253:43757)
0928 08:57:57.716475 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:57.716477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:57.716723 (+ 246us) server_negotiation.cc:407] Connection header received
0928 08:57:57.716821 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:57.716822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:57.716834 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:57.716863 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:57.717262 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39496: BlockingRecv error: recv got EOF from 127.0.0.1:39496 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 08:57:57.727499 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:57:57.726612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39512 (local address 127.23.167.253:43757)
0928 08:57:57.726705 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:57.726707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:57.726949 (+ 242us) server_negotiation.cc:407] Connection header received
0928 08:57:57.727045 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:57.727046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:57.727057 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:57.727085 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:57.727424 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39512: BlockingRecv error: recv got EOF from 127.0.0.1:39512 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 08:57:57.745916 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:57:57.744925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39526 (local address 127.23.167.253:43757)
0928 08:57:57.744970 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:57.744972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:57.745284 (+ 312us) server_negotiation.cc:407] Connection header received
0928 08:57:57.745395 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:57.745397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:57.745411 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:57.745441 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:57.745856 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39526: BlockingRecv error: recv got EOF from 127.0.0.1:39526 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 08:57:57.780326 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:57:57.779420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39528 (local address 127.23.167.253:43757)
0928 08:57:57.779462 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:57.779465 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:57.779720 (+ 255us) server_negotiation.cc:407] Connection header received
0928 08:57:57.779826 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:57.779827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:57.779840 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:57.779871 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:57.780264 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39528: BlockingRecv error: recv got EOF from 127.0.0.1:39528 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 08:57:57.846432 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:57:57.845715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39534 (local address 127.23.167.253:43757)
0928 08:57:57.845794 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:57.845797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:57.845873 (+ 76us) server_negotiation.cc:407] Connection header received
0928 08:57:57.845949 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:57.845951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:57.845962 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:57.846028 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:57.846368 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39534: BlockingRecv error: recv got EOF from 127.0.0.1:39534 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 08:57:57.976257 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:57:57.975562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39544 (local address 127.23.167.253:43757)
0928 08:57:57.975638 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:57.975640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:57.975729 (+ 89us) server_negotiation.cc:407] Connection header received
0928 08:57:57.975801 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:57.975802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:57.975813 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:57.975859 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:57.976194 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39544: BlockingRecv error: recv got EOF from 127.0.0.1:39544 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 08:57:58.234175 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:57:58.233537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39560 (local address 127.23.167.253:43757)
0928 08:57:58.233627 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:58.233629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:58.233652 (+ 23us) server_negotiation.cc:407] Connection header received
0928 08:57:58.233700 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:58.233701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:58.233710 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:58.233754 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:58.234113 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39560: BlockingRecv error: recv got EOF from 127.0.0.1:39560 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 08:57:58.596493 14213 catalog_manager.cc:1259] Loaded cluster ID: b385276130e041f1bac178c70f6c62a2
I20260928 08:57:58.596572 14213 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2fa254d5ce96410d93ac456f67c7ba9d: loading cluster ID for follower catalog manager: success
I20260928 08:57:58.597518 14213 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2fa254d5ce96410d93ac456f67c7ba9d: acquiring CA information for follower catalog manager: success
I20260928 08:57:58.597837 14213 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2fa254d5ce96410d93ac456f67c7ba9d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 08:57:58.657846 14260 catalog_manager.cc:1259] Loaded cluster ID: b385276130e041f1bac178c70f6c62a2
I20260928 08:57:58.657929 14260 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5eb9a512b45c4b9882adfdab958a7482: loading cluster ID for follower catalog manager: success
I20260928 08:57:58.658689 14260 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5eb9a512b45c4b9882adfdab958a7482: acquiring CA information for follower catalog manager: success
I20260928 08:57:58.658950 14260 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5eb9a512b45c4b9882adfdab958a7482: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 08:57:58.749861 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:58.747270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39564 (local address 127.23.167.253:43757)
0928 08:57:58.747389 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:58.747391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:58.747417 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:57:58.747484 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:58.747485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:58.747540 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:58.747611 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:57:58.747784 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:58.748305 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:58.749140 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:58.749340 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:58.749392 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:58.749561 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:57:58.749673 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:57:58.749728 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:58.749756 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 08:57:58.749773 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
I20260928 08:57:58.753830 14333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:58.751523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34476 (local address 127.23.167.252:39927)
0928 08:57:58.751631 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:58.751633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:58.751640 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:57:58.751678 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:58.751678 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:58.751714 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:58.751783 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:57:58.751882 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:58.752368 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:58.753090 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:58.753318 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:58.753371 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:58.753513 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:57:58.753622 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:57:58.753688 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:58.753724 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 08:57:58.753745 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":43,"threads_started":1}
I20260928 08:57:58.757899 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:58.755306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39578 (local address 127.23.167.253:43757)
0928 08:57:58.755346 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:58.755348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:58.755466 (+ 118us) server_negotiation.cc:407] Connection header received
0928 08:57:58.755511 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:58.755512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:58.755522 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:58.755577 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:57:58.755761 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:58.756232 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:58.757181 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:58.757423 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:58.757476 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:58.757639 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:57:58.757674 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:57:58.757719 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:58.757803 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 08:57:58.757827 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 08:57:58.758214 14333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:58.755373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34478 (local address 127.23.167.252:39927)
0928 08:57:58.755426 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:58.755428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:58.755577 (+ 149us) server_negotiation.cc:407] Connection header received
0928 08:57:58.755604 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:58.755605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:58.755614 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:58.755664 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:57:58.755778 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:58.756495 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:58.757483 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:58.757733 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:58.757792 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:58.757914 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:57:58.757948 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:57:58.757984 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:58.758125 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 08:57:58.758144 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 08:57:58.758316 14340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:58.755202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41622 (local address 127.23.167.254:43415)
0928 08:57:58.755396 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:58.755398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:58.755406 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:57:58.755451 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:58.755452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:58.755517 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:58.755592 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:57:58.755737 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:58.756511 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:58.757285 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:58.757568 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:58.757622 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:58.757909 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:57:58.758022 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:57:58.758083 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:58.758217 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 08:57:58.758243 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":57,"threads_started":1}
I20260928 08:57:58.759959 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 13976
I20260928 08:57:58.764636 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 14051
I20260928 08:57:58.769007 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 14126
2026-09-28T08:57:58Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2706 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:57:58 dist-test-slave-pbjp krb5kdc[14344](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:57:58 dist-test-slave-pbjp krb5kdc[14344](info): set up 2 sockets
Sep 28 08:57:58 dist-test-slave-pbjp krb5kdc[14344](info): commencing operation
krb5kdc: starting...
W20260928 08:57:59.725055 24223 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/krb5kdc/test-user.keytab.
Sep 28 08:57:59 dist-test-slave-pbjp krb5kdc[14344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:57:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:57:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.167.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
Entry for principal kudu/127.23.167.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab.
I20260928 08:57:59.836649 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:38589
--webserver_interface=127.23.167.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:38589,127.23.167.253:36821,127.23.167.252:46791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:57:59.915323 14357 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 08:57:59.915570 14357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:57:59.915596 14357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:57:59.915621 14357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:57:59.916766 14357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:57:59.916806 14357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:57:59.916824 14357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:57:59.916841 14357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:57:59.916857 14357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:57:59.918334 14357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:38589,127.23.167.253:36821,127.23.167.252:46791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.254.keytab
--principal=kudu/127.23.167.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.254:38589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.167.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:57:59.918579 14357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:57:59.918787 14357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 08:57:59.923496 14357 server_base.cc:1034] running on GCE node
W20260928 08:57:59.923465 14362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:59.923627 14363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:57:59.923462 14365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:57:59.923974 14357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:57:59.924227 14357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:57:59.925387 14357 hybrid_clock.cc:648] HybridClock initialized: now 1790585879925369 us; error 32 us; skew 500 ppm
Sep 28 08:57:59 dist-test-slave-pbjp krb5kdc[14344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585879, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:57:59.930775 14357 init.cc:377] Logged in from keytab as kudu/127.23.167.254@KRBTEST.COM (short username kudu)
I20260928 08:57:59.931229 14357 webserver.cc:460] Webserver started at http://127.23.167.254:44997/ using document root <none> and password file <none>
I20260928 08:57:59.931456 14357 fs_manager.cc:359] Metadata directory not provided
I20260928 08:57:59.931506 14357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:57:59.931628 14357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:57:59.932690 14357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-0/data/instance:
uuid: "e23a4e5ce22047c5a21a5708750f73bb"
format_stamp: "Formatted at 2026-09-28 08:57:59 on dist-test-slave-pbjp"
server_key: "0f5bb476feb23cb40d59af9b36a1bb40"
server_key_iv: "78ee20f633f416ec08dad900f6a02965"
server_key_version: "encryptionkey@0"
I20260928 08:57:59.933030 14357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-0/wal/instance:
uuid: "e23a4e5ce22047c5a21a5708750f73bb"
format_stamp: "Formatted at 2026-09-28 08:57:59 on dist-test-slave-pbjp"
server_key: "0f5bb476feb23cb40d59af9b36a1bb40"
server_key_iv: "78ee20f633f416ec08dad900f6a02965"
server_key_version: "encryptionkey@0"
I20260928 08:57:59.934345 14357 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:57:59.935009 14372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:57:59.935166 14357 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:57:59.935250 14357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-0/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-0/wal
uuid: "e23a4e5ce22047c5a21a5708750f73bb"
format_stamp: "Formatted at 2026-09-28 08:57:59 on dist-test-slave-pbjp"
server_key: "0f5bb476feb23cb40d59af9b36a1bb40"
server_key_iv: "78ee20f633f416ec08dad900f6a02965"
server_key_version: "encryptionkey@0"
I20260928 08:57:59.935312 14357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:57:59.952965 14357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:57:59.954821 14357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:57:59.954972 14357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:57:59.958835 14357 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.254:38589
I20260928 08:57:59.958873 14424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.254:38589 every 8 connection(s)
I20260928 08:57:59.959219 14357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-0/data/info.pb
I20260928 08:57:59.960134 14425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:57:59.961220 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 14357
I20260928 08:57:59.961319 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-0/wal/instance
I20260928 08:57:59.961524 24223 external_mini_cluster.cc:1428] Setting key 25719e5cd498169e277385b11c8b916a
I20260928 08:57:59.961915 14425 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38589 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:57:59 dist-test-slave-pbjp krb5kdc[14344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585879, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.167.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.253@KRBTEST.COM" created.
I20260928 08:57:59.972358 14428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:59.962590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38589 (local address 127.0.0.1:48978)
0928 08:57:59.962770 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:59.962809 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 08:57:59.963319 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:57:59.963594 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:57:59.963598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:57:59.963954 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:57:59.964185 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:59.964194 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:59.965318 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:59.965319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:59.966461 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:57:59.966465 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:57:59.966929 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:57:59.966931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:57:59.967015 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:59.967534 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:57:59.967545 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:57:59.969158 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:57:59.970859 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:59.970865 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:59.970868 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:59.971060 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:59.971305 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:57:59.971306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:57:59.971306 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:57:59.971391 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:57:59.971757 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:57:59.971763 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:57:59.971835 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 08:57:59.971928 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 08:57:59.972186 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":98,"thread_start_us":81,"threads_started":1}
I20260928 08:57:59.972368 14429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:59.962800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48978 (local address 127.23.167.254:38589)
0928 08:57:59.962917 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 08:57:59.962922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:57:59.962956 (+ 34us) server_negotiation.cc:407] Connection header received
0928 08:57:59.963476 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:57:59.963478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:57:59.963499 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:57:59.963583 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:57:59.964302 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:59.965228 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:59.966543 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:57:59.966848 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:57:59.966923 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:57:59.969234 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:57:59.969243 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:57:59.969243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:57:59.969251 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:57:59.970776 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:59.971133 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:59.971136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:59.971137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:59.971219 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:57:59.971443 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:57:59.971444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:57:59.971444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:57:59.971694 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:57:59.971805 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 08:57:59.971974 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 08:57:59.972096 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":49,"threads_started":1}
I20260928 08:57:59.973256 14425 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.167.253" port: 36821 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:57:59.973656 14428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:59.973522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:36821 (local address 127.0.0.1:37720)
0928 08:57:59.973568 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:59.973623 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.253:36821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 08:57:59.973821 14375 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.167.253:36821: connect: Connection refused (error 111)
I20260928 08:57:59.974270 14428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:57:59.974165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:36821 (local address 127.0.0.1:37736)
0928 08:57:59.974213 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 08:57:59.974245 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.253:36821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 08:57:59.974493 14425 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.167.253:36821: Network error: Client connection negotiation failed: client connection to 127.23.167.253:36821: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.253.keytab.
Entry for principal kudu/127.23.167.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.253.keytab.
I20260928 08:57:59.987634 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.253:36821
--webserver_interface=127.23.167.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.253.keytab
--principal=kudu/127.23.167.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:38589,127.23.167.253:36821,127.23.167.252:46791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:58:00.023770 14425 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.167.253" port: 36821 } attempt: 1
I20260928 08:58:00.024235 14428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.024111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:36821 (local address 127.0.0.1:37748)
0928 08:58:00.024170 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:00.024203 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.253:36821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 08:58:00.024560 14428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.024475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:36821 (local address 127.0.0.1:37760)
0928 08:58:00.024517 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:00.024542 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.253:36821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 08:58:00.024653 14425 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.167.253:36821: Network error: Client connection negotiation failed: client connection to 127.23.167.253:36821: connect: Connection refused (error 111)
W20260928 08:58:00.068233 14432 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 08:58:00.068475 14432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:58:00.068501 14432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:58:00.068527 14432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:58:00.069736 14432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:58:00.069779 14432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:58:00.069797 14432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:58:00.069813 14432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:58:00.069828 14432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:58:00.071372 14432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:38589,127.23.167.253:36821,127.23.167.252:46791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.253.keytab
--principal=kudu/127.23.167.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.253:36821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.167.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:58:00.071635 14432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:58:00.071846 14432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:58:00.076578 14437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:00.076699 14432 server_base.cc:1034] running on GCE node
W20260928 08:58:00.076632 14438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:00.076716 14440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:00.077080 14432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:58:00.077397 14432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:58:00.078544 14432 hybrid_clock.cc:648] HybridClock initialized: now 1790585880078529 us; error 26 us; skew 500 ppm
Sep 28 08:58:00 dist-test-slave-pbjp krb5kdc[14344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585880, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:58:00.081022 14432 init.cc:377] Logged in from keytab as kudu/127.23.167.253@KRBTEST.COM (short username kudu)
I20260928 08:58:00.081502 14432 webserver.cc:460] Webserver started at http://127.23.167.253:39361/ using document root <none> and password file <none>
I20260928 08:58:00.081703 14432 fs_manager.cc:359] Metadata directory not provided
I20260928 08:58:00.081744 14432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:58:00.081835 14432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:58:00.082844 14432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-1/data/instance:
uuid: "ee83f0991eb240e6836da66fb4a37b49"
format_stamp: "Formatted at 2026-09-28 08:58:00 on dist-test-slave-pbjp"
server_key: "42a78a4c55ac425a1746191efbeca7aa"
server_key_iv: "39c2e0fcf909a9b8ff1ad43c4a0ca6fb"
server_key_version: "encryptionkey@0"
I20260928 08:58:00.083137 14432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-1/wal/instance:
uuid: "ee83f0991eb240e6836da66fb4a37b49"
format_stamp: "Formatted at 2026-09-28 08:58:00 on dist-test-slave-pbjp"
server_key: "42a78a4c55ac425a1746191efbeca7aa"
server_key_iv: "39c2e0fcf909a9b8ff1ad43c4a0ca6fb"
server_key_version: "encryptionkey@0"
I20260928 08:58:00.084347 14432 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 08:58:00.084957 14447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:00.085115 14432 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:00.085194 14432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-1/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-1/wal
uuid: "ee83f0991eb240e6836da66fb4a37b49"
format_stamp: "Formatted at 2026-09-28 08:58:00 on dist-test-slave-pbjp"
server_key: "42a78a4c55ac425a1746191efbeca7aa"
server_key_iv: "39c2e0fcf909a9b8ff1ad43c4a0ca6fb"
server_key_version: "encryptionkey@0"
I20260928 08:58:00.085247 14432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:58:00.092787 14425 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.167.253" port: 36821 } attempt: 2
I20260928 08:58:00.093246 14428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.093103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:36821 (local address 127.0.0.1:37768)
0928 08:58:00.093175 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:00.093213 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.253:36821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 08:58:00.093582 14428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.093490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:36821 (local address 127.0.0.1:37774)
0928 08:58:00.093534 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:00.093559 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.253:36821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 08:58:00.093680 14425 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.167.253:36821: Network error: Client connection negotiation failed: client connection to 127.23.167.253:36821: connect: Connection refused (error 111)
I20260928 08:58:00.104588 14432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:58:00.106564 14432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:58:00.106699 14432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:58:00.110584 14432 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.253:36821
I20260928 08:58:00.110617 14499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.253:36821 every 8 connection(s)
I20260928 08:58:00.110967 14432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-1/data/info.pb
I20260928 08:58:00.111840 14500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:00.112129 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 14432
I20260928 08:58:00.112227 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-1/wal/instance
I20260928 08:58:00.112452 24223 external_mini_cluster.cc:1428] Setting key 688da0667f8668703d6c3334d1c68d80
I20260928 08:58:00.113559 14500 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38589 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.167.252@KRBTEST.COM; defaulting to no policy
Sep 28 08:58:00 dist-test-slave-pbjp krb5kdc[14344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585880, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.253@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.167.252@KRBTEST.COM" created.
I20260928 08:58:00.123124 14429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.114415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48984 (local address 127.23.167.254:38589)
0928 08:58:00.114484 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:00.114486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:00.114724 (+ 238us) server_negotiation.cc:407] Connection header received
0928 08:58:00.115199 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:00.115201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:00.115217 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:00.115293 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:00.115920 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.116672 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.117685 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.117989 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.118065 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:00.120299 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:00.120301 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:00.120301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:00.120309 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:00.121753 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:00.122159 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:00.122160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:00.122160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:00.122227 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:00.122493 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:00.122494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:00.122495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:00.122615 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:00.122677 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:00.122985 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 08:58:00.123055 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 08:58:00.123376 14503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.114405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38589 (local address 127.0.0.1:48984)
0928 08:58:00.114586 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:00.114633 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:00.115108 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:00.115347 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:00.115350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:00.115606 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:00.115831 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:00.115836 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.116762 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.116763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:00.117605 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:00.117608 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.118065 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.118066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:00.118164 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:00.118654 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:00.118663 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:00.120196 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:00.121827 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:00.121835 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:00.121837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:00.122087 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:00.122318 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:00.122318 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:00.122319 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:00.122429 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:00.122676 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:00.122683 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:00.122855 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 08:58:00.122973 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 08:58:00.123144 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":95,"threads_started":1}
I20260928 08:58:00.123945 14500 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.167.253" port: 36821 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:58:00 dist-test-slave-pbjp krb5kdc[14344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585880, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.253@KRBTEST.COM for kudu/127.23.167.253@KRBTEST.COM
I20260928 08:58:00.131524 14503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.124208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:36821 (local address 127.0.0.1:37776)
0928 08:58:00.124434 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:00.124445 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:00.124498 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:00.124627 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:00.124628 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:00.124773 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:00.124834 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:00.124835 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.125719 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.125720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:00.126266 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:00.126267 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.126900 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.126901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:00.126983 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:00.127379 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:00.127383 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:00.128647 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:00.130459 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:00.130461 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:00.130462 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:00.130660 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:00.130854 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:00.130855 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:00.130855 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:00.130917 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:00.131192 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:00.131192 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:00.131239 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 08:58:00.131310 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 08:58:00.131392 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209}
I20260928 08:58:00.131603 14505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.124276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37776 (local address 127.23.167.253:36821)
0928 08:58:00.124447 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:00.124451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:00.124493 (+ 42us) server_negotiation.cc:407] Connection header received
0928 08:58:00.124552 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:00.124553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:00.124565 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:00.124622 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:00.124891 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.125645 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.126349 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.126819 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.126898 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:00.128750 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:00.128758 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:00.128758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:00.128765 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:00.130369 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:00.130741 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:00.130744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:00.130744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:00.130799 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:00.130980 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:00.130981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:00.130981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:00.131132 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:00.131195 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:00.131337 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 08:58:00.131443 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":145,"thread_start_us":105,"threads_started":1}
I20260928 08:58:00.132244 14500 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46791 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:58:00.132570 14503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.132463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:46791 (local address 127.0.0.1:35570)
0928 08:58:00.132504 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:00.132548 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.252:46791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 08:58:00.132730 14449 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.167.252:46791: connect: Connection refused (error 111)
I20260928 08:58:00.133164 14503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.133044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:46791 (local address 127.0.0.1:35578)
0928 08:58:00.133117 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:00.133141 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.252:46791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 08:58:00.133271 14500 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.167.252:46791: Network error: Client connection negotiation failed: client connection to 127.23.167.252:46791: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.167.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.252.keytab.
Entry for principal kudu/127.23.167.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.252.keytab.
I20260928 08:58:00.138252 24223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN7pK8W/build/release/bin/kudu
/tmp/dist-test-taskN7pK8W/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.252:46791
--webserver_interface=127.23.167.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.252.keytab
--principal=kudu/127.23.167.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.167.212:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.167.254:38589,127.23.167.253:36821,127.23.167.252:46791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:58:00.182514 14500 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46791 } attempt: 1
I20260928 08:58:00.182999 14503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.182872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:46791 (local address 127.0.0.1:35586)
0928 08:58:00.182933 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:00.182966 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.252:46791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 08:58:00.183374 14503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.183233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:46791 (local address 127.0.0.1:35600)
0928 08:58:00.183295 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:00.183341 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.252:46791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 08:58:00.183516 14500 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.167.252:46791: Network error: Client connection negotiation failed: client connection to 127.23.167.252:46791: connect: Connection refused (error 111)
I20260928 08:58:00.184818 14425 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.167.253" port: 36821 } attempt: 3
Sep 28 08:58:00 dist-test-slave-pbjp krb5kdc[14344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585879, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for kudu/127.23.167.253@KRBTEST.COM
I20260928 08:58:00.191604 14505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.185196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37786 (local address 127.23.167.253:36821)
0928 08:58:00.185250 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:00.185252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:00.185262 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:58:00.185352 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:00.185353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:00.185362 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:00.185453 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:00.185768 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.186265 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.186969 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.187204 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.187260 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:00.189345 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:00.189346 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:00.189346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:00.189353 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:00.190615 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:00.190973 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:00.190973 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:00.190974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:00.191018 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:00.191178 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:00.191178 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:00.191179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:00.191274 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:00.191315 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:00.191470 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 08:58:00.191541 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 08:58:00.191641 14428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.185160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:36821 (local address 127.0.0.1:37786)
0928 08:58:00.185198 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:00.185209 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:00.185278 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:00.185479 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:00.185481 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:00.185631 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:00.185699 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:00.185701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.186335 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.186336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:00.186871 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:00.186873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.187266 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.187267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:00.187337 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:00.187794 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:00.187799 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:00.189257 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:00.190692 (+ 1435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:00.190693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:00.190694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:00.190909 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:00.191067 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:00.191068 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:00.191068 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:00.191129 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:00.191321 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:00.191322 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:00.191373 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 08:58:00.191443 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 08:58:00.191542 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 08:58:00.192031 14425 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46791 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:58:00.192340 14428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.192245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:46791 (local address 127.0.0.1:35604)
0928 08:58:00.192288 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:00.192319 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.252:46791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 08:58:00.192617 14428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.192540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:46791 (local address 127.0.0.1:35620)
0928 08:58:00.192577 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:00.192600 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.252:46791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 08:58:00.192706 14425 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.167.252:46791: Network error: Client connection negotiation failed: client connection to 127.23.167.252:46791: connect: Connection refused (error 111)
W20260928 08:58:00.220306 14507 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 08:58:00.220530 14507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:58:00.220546 14507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:58:00.220566 14507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:58:00.221801 14507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:58:00.221844 14507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:58:00.221858 14507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:58:00.221868 14507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:58:00.221881 14507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:58:00.223373 14507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.167.212:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.167.254:38589,127.23.167.253:36821,127.23.167.252:46791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/krb5kdc/kudu_127.23.167.252.keytab
--principal=kudu/127.23.167.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.167.252:46791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.167.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:58:00.223610 14507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:58:00.223829 14507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 08:58:00.223866 14425 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46791 } attempt: 1
I20260928 08:58:00.224319 14428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.224162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:46791 (local address 127.0.0.1:35634)
0928 08:58:00.224231 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:00.224282 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.252:46791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 08:58:00.224680 14428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.224592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:46791 (local address 127.0.0.1:35642)
0928 08:58:00.224631 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:00.224659 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.252:46791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 08:58:00.224768 14425 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.167.252:46791: Network error: Client connection negotiation failed: client connection to 127.23.167.252:46791: connect: Connection refused (error 111)
W20260928 08:58:00.228587 14512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:00.228741 14507 server_base.cc:1034] running on GCE node
W20260928 08:58:00.228606 14513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:00.228617 14515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:00.229038 14507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:58:00.229332 14507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:58:00.230509 14507 hybrid_clock.cc:648] HybridClock initialized: now 1790585880230491 us; error 26 us; skew 500 ppm
Sep 28 08:58:00 dist-test-slave-pbjp krb5kdc[14344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585880, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:58:00.232964 14507 init.cc:377] Logged in from keytab as kudu/127.23.167.252@KRBTEST.COM (short username kudu)
I20260928 08:58:00.233527 14507 webserver.cc:460] Webserver started at http://127.23.167.252:37569/ using document root <none> and password file <none>
I20260928 08:58:00.233798 14507 fs_manager.cc:359] Metadata directory not provided
I20260928 08:58:00.233862 14507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:58:00.233994 14507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:58:00.235127 14507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-2/data/instance:
uuid: "7f358adbe9764bea8a557f258f1ae61e"
format_stamp: "Formatted at 2026-09-28 08:58:00 on dist-test-slave-pbjp"
server_key: "4bf15f06db0452d515f062d72e3e25f6"
server_key_iv: "4962136342c90f377381977b7ebc35d2"
server_key_version: "encryptionkey@0"
I20260928 08:58:00.235467 14507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-2/wal/instance:
uuid: "7f358adbe9764bea8a557f258f1ae61e"
format_stamp: "Formatted at 2026-09-28 08:58:00 on dist-test-slave-pbjp"
server_key: "4bf15f06db0452d515f062d72e3e25f6"
server_key_iv: "4962136342c90f377381977b7ebc35d2"
server_key_version: "encryptionkey@0"
I20260928 08:58:00.236763 14507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 08:58:00.237583 14522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:00.237733 14507 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:00.237813 14507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-2/data,/tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-2/wal
uuid: "7f358adbe9764bea8a557f258f1ae61e"
format_stamp: "Formatted at 2026-09-28 08:58:00 on dist-test-slave-pbjp"
server_key: "4bf15f06db0452d515f062d72e3e25f6"
server_key_iv: "4962136342c90f377381977b7ebc35d2"
server_key_version: "encryptionkey@0"
I20260928 08:58:00.237898 14507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:58:00.251672 14500 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46791 } attempt: 2
I20260928 08:58:00.252147 14503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.252019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:46791 (local address 127.0.0.1:35650)
0928 08:58:00.252078 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:00.252112 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.252:46791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 08:58:00.252507 14503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.252411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:46791 (local address 127.0.0.1:35664)
0928 08:58:00.252456 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:00.252484 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.167.252:46791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 08:58:00.252593 14500 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.167.252:46791: Network error: Client connection negotiation failed: client connection to 127.23.167.252:46791: connect: Connection refused (error 111)
I20260928 08:58:00.255163 14507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:58:00.257011 14507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:58:00.257174 14507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:58:00.260867 14507 rpc_server.cc:307] RPC server started. Bound to: 127.23.167.252:46791
I20260928 08:58:00.260907 14574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.167.252:46791 every 8 connection(s)
I20260928 08:58:00.261265 14507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-2/data/info.pb
I20260928 08:58:00.262120 14575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:00.262859 24223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN7pK8W/build/release/bin/kudu as pid 14507
I20260928 08:58:00.262949 24223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN7pK8W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585589999956-24223-0/minicluster-data/master-2/wal/instance
I20260928 08:58:00.263144 24223 external_mini_cluster.cc:1428] Setting key 61db752cf12e78ff3fda48fd04140fdc
I20260928 08:58:00.263792 14575 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38589 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:58:00 dist-test-slave-pbjp krb5kdc[14344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585880, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.252@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
Sep 28 08:58:00 dist-test-slave-pbjp krb5kdc[14344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.252@KRBTEST.COM
I20260928 08:58:00.271857 14429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.264765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48986 (local address 127.23.167.254:38589)
0928 08:58:00.264819 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:00.264821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:00.264911 (+ 90us) server_negotiation.cc:407] Connection header received
0928 08:58:00.264956 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:00.264957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:00.264983 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:00.265054 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:00.265537 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.265999 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.266834 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.267041 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.267097 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:00.269212 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:00.269214 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:00.269214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:00.269222 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:00.270512 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:00.270901 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:00.270902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:00.270903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:00.270955 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:00.271172 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:00.271173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:00.271173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:00.271293 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:00.271347 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:00.271671 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 08:58:00.271759 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 08:58:00.272001 14579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.264702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.254:38589 (local address 127.0.0.1:48986)
0928 08:58:00.264817 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:00.264855 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:00.264912 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:00.265102 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:00.265105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:00.265340 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:00.265449 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:00.265453 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.266071 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.266072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:00.266757 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:00.266759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.267109 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.267110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:00.267200 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:00.267621 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:00.267629 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:00.269135 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:00.270585 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:00.270599 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:00.270601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:00.270815 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:00.271009 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:00.271010 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:00.271011 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:00.271111 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:00.271345 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:00.271352 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:00.271530 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 08:58:00.271625 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 08:58:00.271778 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":67,"threads_started":1}
I20260928 08:58:00.272569 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.263644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35668 (local address 127.23.167.252:46791)
0928 08:58:00.263787 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:00.263795 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:00.263822 (+ 27us) server_negotiation.cc:407] Connection header received
0928 08:58:00.264287 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:00.264294 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:00.264345 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:00.264521 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:00.265247 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.266099 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.266800 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.267124 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.267204 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:00.269679 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:00.269686 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:00.269686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:00.269693 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:00.271139 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:00.271449 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:00.271452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:00.271452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:00.271500 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:00.271667 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:00.271668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:00.271668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:00.271796 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:00.271922 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:00.272391 (+ 469us) server_negotiation.cc:299] Negotiation successful
0928 08:58:00.272467 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
I20260928 08:58:00.272574 14575 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.167.253" port: 36821 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:58:00 dist-test-slave-pbjp krb5kdc[14344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585880, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.252@KRBTEST.COM for kudu/127.23.167.253@KRBTEST.COM
Sep 28 08:58:00 dist-test-slave-pbjp krb5kdc[14344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.254@KRBTEST.COM
I20260928 08:58:00.279925 14579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.272933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.253:36821 (local address 127.0.0.1:37798)
0928 08:58:00.273085 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:00.273094 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:00.273133 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:00.273280 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:00.273281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:00.273434 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:00.273488 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:00.273489 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.274299 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.274300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:00.275056 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:00.275059 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.275507 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.275508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:00.275580 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:00.275993 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:00.275997 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:00.277397 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:00.279011 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:00.279012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:00.279012 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:00.279175 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:00.279351 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:00.279352 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:00.279352 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:00.279408 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:00.279621 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:00.279622 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:00.279667 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 08:58:00.279727 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 08:58:00.279824 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
I20260928 08:58:00.279925 14505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.272959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37798 (local address 127.23.167.253:36821)
0928 08:58:00.273024 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:00.273026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:00.273121 (+ 95us) server_negotiation.cc:407] Connection header received
0928 08:58:00.273181 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:00.273182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:00.273192 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:00.273235 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:00.273564 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.274209 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.275118 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.275427 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.275490 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:00.277476 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:00.277478 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:00.277478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:00.277487 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:00.278921 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:00.279233 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:00.279234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:00.279235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:00.279301 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:00.279457 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:00.279458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:00.279458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:00.279570 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:00.279623 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:00.279755 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 08:58:00.279824 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 08:58:00.280539 14575 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46791 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:58:00.281042 14429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.273287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48990 (local address 127.23.167.254:38589)
0928 08:58:00.273324 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:00.273326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:00.273364 (+ 38us) server_negotiation.cc:407] Connection header received
0928 08:58:00.273419 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:00.273420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:00.273429 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:00.273473 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:00.274021 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.274636 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.275505 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.275783 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.275846 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:00.277954 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:00.277955 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:00.277955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:00.277962 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:00.279241 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:00.279564 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:00.279566 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:00.279566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:00.279619 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:00.279772 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:00.279773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:00.279773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:00.279877 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:00.280870 (+ 993us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:00.280912 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 08:58:00.280980 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 28 08:58:00 dist-test-slave-pbjp krb5kdc[14344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585880, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.252@KRBTEST.COM for kudu/127.23.167.252@KRBTEST.COM
Sep 28 08:58:00 dist-test-slave-pbjp krb5kdc[14344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.167.253@KRBTEST.COM
I20260928 08:58:00.286878 14579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.280778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:46791 (local address 127.0.0.1:35682)
0928 08:58:00.280820 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:00.280830 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:00.280865 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:00.280955 (+ 90us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:00.280956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:00.281117 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:00.281152 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:00.281154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.281869 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.281870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:00.282421 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:00.282423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.282784 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.282785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:00.282836 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:00.283184 (+ 348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:00.283189 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:00.284554 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:00.285996 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:00.285997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:00.285997 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:00.286149 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:00.286334 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:00.286335 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:00.286335 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:00.286381 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:00.286576 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:00.286577 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:00.286612 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 08:58:00.286668 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 08:58:00.286762 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 08:58:00.286927 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.280845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35682 (local address 127.23.167.252:46791)
0928 08:58:00.280878 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:00.280879 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:00.280886 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:58:00.280904 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:00.280904 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:00.280913 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:00.280955 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:00.281218 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.281807 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.282517 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.282721 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.282779 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:00.284632 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:00.284633 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:00.284633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:00.284639 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:00.285906 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:00.286235 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:00.286236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:00.286236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:00.286282 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:00.286431 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:00.286431 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:00.286432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:00.286526 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:00.286578 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:00.286693 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 08:58:00.286811 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":19}
I20260928 08:58:00.288184 14505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.281635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37814 (local address 127.23.167.253:36821)
0928 08:58:00.281667 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:00.281669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:00.281677 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:58:00.281761 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:00.281761 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:00.281770 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:00.281817 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:00.282243 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.282772 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.283400 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.283688 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.283739 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:00.285812 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:00.285814 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:00.285814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:00.285821 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:00.287186 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:00.287535 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:00.287535 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:00.287536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:00.287585 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:00.287748 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:00.287749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:00.287749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:00.287856 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:00.288012 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:00.288065 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 08:58:00.288122 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260928 08:58:00.288597 14575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f358adbe9764bea8a557f258f1ae61e: Bootstrap starting.
I20260928 08:58:00.289592 24223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:58:00.289630 14575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f358adbe9764bea8a557f258f1ae61e: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:00.289881 14575 log.cc:824] T 00000000000000000000000000000000 P 7f358adbe9764bea8a557f258f1ae61e: Log is configured to *not* fsync() on all Append() calls
I20260928 08:58:00.290745 14575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f358adbe9764bea8a557f258f1ae61e: No bootstrap required, opened a new log
I20260928 08:58:00.290386 14399 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e23a4e5ce22047c5a21a5708750f73bb"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48990
I20260928 08:58:00.292651 14575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f358adbe9764bea8a557f258f1ae61e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23a4e5ce22047c5a21a5708750f73bb" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38589 } } peers { permanent_uuid: "ee83f0991eb240e6836da66fb4a37b49" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 36821 } } peers { permanent_uuid: "7f358adbe9764bea8a557f258f1ae61e" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46791 } }
I20260928 08:58:00.292809 14575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f358adbe9764bea8a557f258f1ae61e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f358adbe9764bea8a557f258f1ae61e, State: Initialized, Role: FOLLOWER
I20260928 08:58:00.292892 14575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f358adbe9764bea8a557f258f1ae61e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23a4e5ce22047c5a21a5708750f73bb" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38589 } } peers { permanent_uuid: "ee83f0991eb240e6836da66fb4a37b49" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 36821 } } peers { permanent_uuid: "7f358adbe9764bea8a557f258f1ae61e" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46791 } }
I20260928 08:58:00.293203 14581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f358adbe9764bea8a557f258f1ae61e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23a4e5ce22047c5a21a5708750f73bb" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38589 } } peers { permanent_uuid: "ee83f0991eb240e6836da66fb4a37b49" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 36821 } } peers { permanent_uuid: "7f358adbe9764bea8a557f258f1ae61e" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46791 } } }
I20260928 08:58:00.293259 14575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f358adbe9764bea8a557f258f1ae61e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:58:00.293301 14581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f358adbe9764bea8a557f258f1ae61e [sys.catalog]: This master's current role is: FOLLOWER
W20260928 08:58:00.295037 14592 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7f358adbe9764bea8a557f258f1ae61e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:58:00.295094 14592 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:58:00.295238 14507 master_runner.cc:186] Error getting master registration for 127.23.167.254:38589: OK, instance_id { permanent_uuid: "e23a4e5ce22047c5a21a5708750f73bb" instance_seqno: 1790585879957749 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 08:58:00.297945 14507 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 08:58:00.299904 14425 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46791 } attempt: 2
Sep 28 08:58:00 dist-test-slave-pbjp krb5kdc[14344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585879, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.254@KRBTEST.COM for kudu/127.23.167.252@KRBTEST.COM
I20260928 08:58:00.307858 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.300386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35694 (local address 127.23.167.252:46791)
0928 08:58:00.300440 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:00.300441 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:00.300450 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:58:00.300471 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:00.300472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:00.300482 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:00.300537 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:00.300866 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.301658 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.302641 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.302942 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.303015 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:00.305181 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:00.305183 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:00.305183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:00.305191 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:00.306690 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:00.307072 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:00.307073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:00.307073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:00.307132 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:00.307296 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:00.307297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:00.307297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:00.307475 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:00.307545 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:00.307681 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 08:58:00.307761 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 08:58:00.307860 14428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.300166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:46791 (local address 127.0.0.1:35694)
0928 08:58:00.300227 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:00.300241 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:00.300296 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:00.300557 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:00.300558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:00.300745 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:00.300796 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:00.300799 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.301736 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.301738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:00.302559 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:00.302562 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.303005 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.303006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:00.303069 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:00.303547 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:00.303552 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:00.305056 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:00.306770 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:00.306771 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:00.306772 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:00.307007 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:00.307188 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:00.307189 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:00.307189 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:00.307246 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:00.307539 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:00.307540 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:00.307588 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 08:58:00.307648 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 08:58:00.307752 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 08:58:00.309548 14425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e23a4e5ce22047c5a21a5708750f73bb: Bootstrap starting.
I20260928 08:58:00.310300 14425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e23a4e5ce22047c5a21a5708750f73bb: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:00.310607 14425 log.cc:824] T 00000000000000000000000000000000 P e23a4e5ce22047c5a21a5708750f73bb: Log is configured to *not* fsync() on all Append() calls
I20260928 08:58:00.311472 14425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e23a4e5ce22047c5a21a5708750f73bb: No bootstrap required, opened a new log
I20260928 08:58:00.312817 14425 raft_consensus.cc:348] T 00000000000000000000000000000000 P e23a4e5ce22047c5a21a5708750f73bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23a4e5ce22047c5a21a5708750f73bb" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38589 } } peers { permanent_uuid: "ee83f0991eb240e6836da66fb4a37b49" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 36821 } } peers { permanent_uuid: "7f358adbe9764bea8a557f258f1ae61e" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46791 } }
I20260928 08:58:00.312975 14425 raft_consensus.cc:729] T 00000000000000000000000000000000 P e23a4e5ce22047c5a21a5708750f73bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e23a4e5ce22047c5a21a5708750f73bb, State: Initialized, Role: FOLLOWER
I20260928 08:58:00.313058 14425 consensus_queue.cc:260] T 00000000000000000000000000000000 P e23a4e5ce22047c5a21a5708750f73bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23a4e5ce22047c5a21a5708750f73bb" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38589 } } peers { permanent_uuid: "ee83f0991eb240e6836da66fb4a37b49" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 36821 } } peers { permanent_uuid: "7f358adbe9764bea8a557f258f1ae61e" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46791 } }
I20260928 08:58:00.313392 14425 sys_catalog.cc:564] T 00000000000000000000000000000000 P e23a4e5ce22047c5a21a5708750f73bb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:58:00.313346 14595 sys_catalog.cc:455] T 00000000000000000000000000000000 P e23a4e5ce22047c5a21a5708750f73bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23a4e5ce22047c5a21a5708750f73bb" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38589 } } peers { permanent_uuid: "ee83f0991eb240e6836da66fb4a37b49" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 36821 } } peers { permanent_uuid: "7f358adbe9764bea8a557f258f1ae61e" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46791 } } }
I20260928 08:58:00.313417 14399 raft_consensus.cc:482] T 00000000000000000000000000000000 P e23a4e5ce22047c5a21a5708750f73bb [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 08:58:00.313491 14595 sys_catalog.cc:458] T 00000000000000000000000000000000 P e23a4e5ce22047c5a21a5708750f73bb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:58:00.313561 14399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e23a4e5ce22047c5a21a5708750f73bb [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:00.314255 14399 raft_consensus.cc:504] T 00000000000000000000000000000000 P e23a4e5ce22047c5a21a5708750f73bb [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23a4e5ce22047c5a21a5708750f73bb" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38589 } } peers { permanent_uuid: "ee83f0991eb240e6836da66fb4a37b49" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 36821 } } peers { permanent_uuid: "7f358adbe9764bea8a557f258f1ae61e" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46791 } }
I20260928 08:58:00.314492 14399 leader_election.cc:288] T 00000000000000000000000000000000 P e23a4e5ce22047c5a21a5708750f73bb [CANDIDATE]: Term 1 election: Requested vote from peers ee83f0991eb240e6836da66fb4a37b49 (127.23.167.253:36821), 7f358adbe9764bea8a557f258f1ae61e (127.23.167.252:46791)
I20260928 08:58:00.314841 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e23a4e5ce22047c5a21a5708750f73bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7f358adbe9764bea8a557f258f1ae61e"
I20260928 08:58:00.314975 14549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f358adbe9764bea8a557f258f1ae61e [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:00.315837 14549 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7f358adbe9764bea8a557f258f1ae61e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e23a4e5ce22047c5a21a5708750f73bb in term 1.
I20260928 08:58:00.316058 14374 leader_election.cc:302] T 00000000000000000000000000000000 P e23a4e5ce22047c5a21a5708750f73bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f358adbe9764bea8a557f258f1ae61e, e23a4e5ce22047c5a21a5708750f73bb; no voters:
I20260928 08:58:00.316656 14595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e23a4e5ce22047c5a21a5708750f73bb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:58:00.316803 14595 raft_consensus.cc:686] T 00000000000000000000000000000000 P e23a4e5ce22047c5a21a5708750f73bb [term 1 LEADER]: Becoming Leader. State: Replica: e23a4e5ce22047c5a21a5708750f73bb, State: Running, Role: LEADER
I20260928 08:58:00.316888 14595 consensus_queue.cc:237] T 00000000000000000000000000000000 P e23a4e5ce22047c5a21a5708750f73bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23a4e5ce22047c5a21a5708750f73bb" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38589 } } peers { permanent_uuid: "ee83f0991eb240e6836da66fb4a37b49" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 36821 } } peers { permanent_uuid: "7f358adbe9764bea8a557f258f1ae61e" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46791 } }
I20260928 08:58:00.318081 14612 sys_catalog.cc:455] T 00000000000000000000000000000000 P e23a4e5ce22047c5a21a5708750f73bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e23a4e5ce22047c5a21a5708750f73bb. Latest consensus state: current_term: 1 leader_uuid: "e23a4e5ce22047c5a21a5708750f73bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23a4e5ce22047c5a21a5708750f73bb" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38589 } } peers { permanent_uuid: "ee83f0991eb240e6836da66fb4a37b49" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 36821 } } peers { permanent_uuid: "7f358adbe9764bea8a557f258f1ae61e" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46791 } } }
I20260928 08:58:00.318256 14612 sys_catalog.cc:458] T 00000000000000000000000000000000 P e23a4e5ce22047c5a21a5708750f73bb [sys.catalog]: This master's current role is: LEADER
W20260928 08:58:00.318312 14606 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e23a4e5ce22047c5a21a5708750f73bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:58:00.318361 14606 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:58:00.318070 14357 master_runner.cc:186] Error getting master registration for 127.23.167.253:36821: OK, instance_id { permanent_uuid: "ee83f0991eb240e6836da66fb4a37b49" instance_seqno: 1790585880109484 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 08:58:00.320245 14357 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 08:58:00.321846 14615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:58:00.322093 14615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:58:00.326023 14549 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7f358adbe9764bea8a557f258f1ae61e [term 1 FOLLOWER]: Refusing update from remote peer e23a4e5ce22047c5a21a5708750f73bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:58:00.326414 14612 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e23a4e5ce22047c5a21a5708750f73bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f358adbe9764bea8a557f258f1ae61e" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:58:00.327775 14581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f358adbe9764bea8a557f258f1ae61e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e23a4e5ce22047c5a21a5708750f73bb. Latest consensus state: current_term: 1 leader_uuid: "e23a4e5ce22047c5a21a5708750f73bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23a4e5ce22047c5a21a5708750f73bb" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38589 } } peers { permanent_uuid: "ee83f0991eb240e6836da66fb4a37b49" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 36821 } } peers { permanent_uuid: "7f358adbe9764bea8a557f258f1ae61e" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46791 } } }
I20260928 08:58:00.327860 14581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f358adbe9764bea8a557f258f1ae61e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:58:00.329444 14612 sys_catalog.cc:455] T 00000000000000000000000000000000 P e23a4e5ce22047c5a21a5708750f73bb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e23a4e5ce22047c5a21a5708750f73bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23a4e5ce22047c5a21a5708750f73bb" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38589 } } peers { permanent_uuid: "ee83f0991eb240e6836da66fb4a37b49" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 36821 } } peers { permanent_uuid: "7f358adbe9764bea8a557f258f1ae61e" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46791 } } }
I20260928 08:58:00.329530 14612 sys_catalog.cc:458] T 00000000000000000000000000000000 P e23a4e5ce22047c5a21a5708750f73bb [sys.catalog]: This master's current role is: LEADER
I20260928 08:58:00.329872 14581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f358adbe9764bea8a557f258f1ae61e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e23a4e5ce22047c5a21a5708750f73bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23a4e5ce22047c5a21a5708750f73bb" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38589 } } peers { permanent_uuid: "ee83f0991eb240e6836da66fb4a37b49" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 36821 } } peers { permanent_uuid: "7f358adbe9764bea8a557f258f1ae61e" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46791 } } }
I20260928 08:58:00.329950 14581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f358adbe9764bea8a557f258f1ae61e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:58:00.330215 14618 mvcc.cc:205] Tried to move back new op lower bound from 7334239765812690944 to 7334239765778796544. Current Snapshot: MvccSnapshot[applied={T|T < 7334239765812690944}]
I20260928 08:58:00.330345 14619 mvcc.cc:205] Tried to move back new op lower bound from 7334239765812690944 to 7334239765778796544. Current Snapshot: MvccSnapshot[applied={T|T < 7334239765812690944}]
I20260928 08:58:00.330580 14429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.318618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49000 (local address 127.23.167.254:38589)
0928 08:58:00.318654 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:00.318656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:00.321410 (+ 2754us) server_negotiation.cc:407] Connection header received
0928 08:58:00.321495 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:00.321496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:00.321506 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:00.321534 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:00.322573 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.323313 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.324867 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.325195 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.325236 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:00.327433 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:00.327434 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:00.327435 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:00.327442 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:00.328966 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:00.329499 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:00.329500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:00.329500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:00.329556 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:00.329858 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:00.329859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:00.329859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:00.330006 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:00.330080 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:00.330401 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 08:58:00.330488 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 08:58:00.331423 14615 catalog_manager.cc:1347] Generated new cluster ID: 1da916ea05f74f42a0e0034fa3589a31
I20260928 08:58:00.331468 14615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:58:00.333904 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.318781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35706 (local address 127.23.167.252:46791)
0928 08:58:00.318812 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:00.318814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:00.326063 (+ 7249us) server_negotiation.cc:407] Connection header received
0928 08:58:00.326166 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:00.326167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:00.326179 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:00.326208 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:00.326844 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.327516 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.329018 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.329317 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.329360 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:00.330828 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:00.330829 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:00.330830 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:00.330837 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:00.332375 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:00.333120 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:00.333121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:00.333122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:00.333177 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:00.333369 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:00.333370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:00.333370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:00.333489 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:00.333552 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:00.333744 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 08:58:00.333820 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 08:58:00.339128 14505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.315694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37816 (local address 127.23.167.253:36821)
0928 08:58:00.315737 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:00.315740 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:00.321239 (+ 5499us) server_negotiation.cc:407] Connection header received
0928 08:58:00.321349 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:00.321351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:00.321364 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:00.321431 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:00.331258 (+ 9827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.331966 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.333314 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.333619 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.333658 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:00.335860 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:00.335862 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:00.335862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:00.335870 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:00.337426 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:00.338034 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:00.338036 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:00.338036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:00.338108 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:00.338401 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:00.338402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:00.338403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:00.338557 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:00.338636 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:00.338953 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 08:58:00.339034 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 08:58:00.343714 14500 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46791 } attempt: 3
Sep 28 08:58:00 dist-test-slave-pbjp krb5kdc[14344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585880, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.167.253@KRBTEST.COM for kudu/127.23.167.252@KRBTEST.COM
I20260928 08:58:00.352924 14503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.343975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.167.252:46791 (local address 127.0.0.1:35714)
0928 08:58:00.344053 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:00.344063 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:00.344097 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:00.344310 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:00.344311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:00.344591 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:00.344637 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:00.344640 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.346170 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.346170 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:00.347042 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:00.347045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.347597 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.347598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:00.347667 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:00.348126 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:00.348131 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:00.349617 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:00.351753 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:00.351754 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:00.351754 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:00.351972 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:00.352214 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:00.352215 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:00.352215 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:00.352284 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:00.352605 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:00.352606 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:00.352665 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 08:58:00.352724 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 08:58:00.352819 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 08:58:00.353353 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:00.344164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35714 (local address 127.23.167.252:46791)
0928 08:58:00.344205 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:00.344206 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:00.344213 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:58:00.344234 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:00.344234 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:00.344244 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:00.344287 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:00.344773 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.346049 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.347148 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:00.347473 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:00.347529 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:00.349721 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:00.349723 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:00.349723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:00.349731 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:00.351548 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:00.352064 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:00.352066 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:00.352066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:00.352138 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:00.352356 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:00.352356 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:00.352357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:00.352508 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:00.352563 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:00.353185 (+ 622us) server_negotiation.cc:299] Negotiation successful
0928 08:58:00.353261 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 08:58:00.354856 14500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee83f0991eb240e6836da66fb4a37b49: Bootstrap starting.
I20260928 08:58:00.355633 14500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee83f0991eb240e6836da66fb4a37b49: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:00.355914 14500 log.cc:824] T 00000000000000000000000000000000 P ee83f0991eb240e6836da66fb4a37b49: Log is configured to *not* fsync() on all Append() calls
I20260928 08:58:00.356834 14615 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:58:00.357402 14500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee83f0991eb240e6836da66fb4a37b49: No bootstrap required, opened a new log
I20260928 08:58:00.358121 14615 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:58:00.359484 14500 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee83f0991eb240e6836da66fb4a37b49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23a4e5ce22047c5a21a5708750f73bb" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38589 } } peers { permanent_uuid: "ee83f0991eb240e6836da66fb4a37b49" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 36821 } } peers { permanent_uuid: "7f358adbe9764bea8a557f258f1ae61e" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46791 } }
I20260928 08:58:00.359628 14500 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee83f0991eb240e6836da66fb4a37b49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee83f0991eb240e6836da66fb4a37b49, State: Initialized, Role: FOLLOWER
I20260928 08:58:00.359717 14500 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee83f0991eb240e6836da66fb4a37b49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23a4e5ce22047c5a21a5708750f73bb" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38589 } } peers { permanent_uuid: "ee83f0991eb240e6836da66fb4a37b49" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 36821 } } peers { permanent_uuid: "7f358adbe9764bea8a557f258f1ae61e" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46791 } }
I20260928 08:58:00.360049 14500 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee83f0991eb240e6836da66fb4a37b49 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:58:00.360129 14473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee83f0991eb240e6836da66fb4a37b49 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:00.360988 14473 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ee83f0991eb240e6836da66fb4a37b49 [term 1 FOLLOWER]: Refusing update from remote peer e23a4e5ce22047c5a21a5708750f73bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:58:00.361110 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e23a4e5ce22047c5a21a5708750f73bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ee83f0991eb240e6836da66fb4a37b49"
I20260928 08:58:00.361308 14612 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e23a4e5ce22047c5a21a5708750f73bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee83f0991eb240e6836da66fb4a37b49" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 36821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:58:00.361879 14474 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ee83f0991eb240e6836da66fb4a37b49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e23a4e5ce22047c5a21a5708750f73bb in term 1.
W20260928 08:58:00.364383 14639 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee83f0991eb240e6836da66fb4a37b49: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:58:00.364451 14639 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:58:00.362994 14624 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee83f0991eb240e6836da66fb4a37b49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e23a4e5ce22047c5a21a5708750f73bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23a4e5ce22047c5a21a5708750f73bb" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38589 } } peers { permanent_uuid: "ee83f0991eb240e6836da66fb4a37b49" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 36821 } } peers { permanent_uuid: "7f358adbe9764bea8a557f258f1ae61e" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46791 } } }
I20260928 08:58:00.365362 14624 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee83f0991eb240e6836da66fb4a37b49 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:58:00.365433 14624 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee83f0991eb240e6836da66fb4a37b49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e23a4e5ce22047c5a21a5708750f73bb. Latest consensus state: current_term: 1 leader_uuid: "e23a4e5ce22047c5a21a5708750f73bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23a4e5ce22047c5a21a5708750f73bb" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38589 } } peers { permanent_uuid: "ee83f0991eb240e6836da66fb4a37b49" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 36821 } } peers { permanent_uuid: "7f358adbe9764bea8a557f258f1ae61e" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46791 } } }
I20260928 08:58:00.365499 14624 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee83f0991eb240e6836da66fb4a37b49 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:58:00.365527 14624 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee83f0991eb240e6836da66fb4a37b49 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e23a4e5ce22047c5a21a5708750f73bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23a4e5ce22047c5a21a5708750f73bb" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38589 } } peers { permanent_uuid: "ee83f0991eb240e6836da66fb4a37b49" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 36821 } } peers { permanent_uuid: "7f358adbe9764bea8a557f258f1ae61e" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46791 } } }
I20260928 08:58:00.365578 14624 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee83f0991eb240e6836da66fb4a37b49 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:58:00.369644 14636 mvcc.cc:205] Tried to move back new op lower bound from 7334239765936840704 to 7334239765778796544. Current Snapshot: MvccSnapshot[applied={T|T < 7334239765812690944}]
I20260928 08:58:00.372640 14612 sys_catalog.cc:455] T 00000000000000000000000000000000 P e23a4e5ce22047c5a21a5708750f73bb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e23a4e5ce22047c5a21a5708750f73bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23a4e5ce22047c5a21a5708750f73bb" member_type: VOTER last_known_addr { host: "127.23.167.254" port: 38589 } } peers { permanent_uuid: "ee83f0991eb240e6836da66fb4a37b49" member_type: VOTER last_known_addr { host: "127.23.167.253" port: 36821 } } peers { permanent_uuid: "7f358adbe9764bea8a557f258f1ae61e" member_type: VOTER last_known_addr { host: "127.23.167.252" port: 46791 } } }
I20260928 08:58:00.372763 14612 sys_catalog.cc:458] T 00000000000000000000000000000000 P e23a4e5ce22047c5a21a5708750f73bb [sys.catalog]: This master's current role is: LEADER
I20260928 08:58:00.375383 14615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e23a4e5ce22047c5a21a5708750f73bb: Generated new TSK 0
I20260928 08:58:00.375622 14615 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 08:58:00.410389 14505 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:58:00.408606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37820 (local address 127.23.167.253:36821)
0928 08:58:00.408655 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:00.408657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:00.409686 (+ 1029us) server_negotiation.cc:407] Connection header received
0928 08:58:00.409796 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:00.409797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:00.409809 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:00.409840 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:00.410312 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37820: BlockingRecv error: recv got EOF from 127.0.0.1:37820 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 08:58:00.411121 14429 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:58:00.409042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49014 (local address 127.23.167.254:38589)
0928 08:58:00.409111 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:00.409113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:00.410642 (+ 1529us) server_negotiation.cc:407] Connection header received
0928 08:58:00.410732 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:00.410733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:00.410744 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:00.410770 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:00.411057 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49014: BlockingRecv error: recv got EOF from 127.0.0.1:49014 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 08:58:00.411787 14578 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:58:00.409397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35722 (local address 127.23.167.252:46791)
0928 08:58:00.409441 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:00.409443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:00.411339 (+ 1896us) server_negotiation.cc:407] Connection header received
0928 08:58:00.411425 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:00.411426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:00.411437 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:00.411463 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:00.411723 (+ 260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35722: BlockingRecv error: recv got EOF from 127.0.0.1:35722 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 08:58:00.414103 14505 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:58:00.412977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37832 (local address 127.23.167.253:36821)
0928 08:58:00.413013 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:00.413015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:00.413255 (+ 240us) server_negotiation.cc:407] Connection header received
0928 08:58:00.413369 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:00.413370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:00.413390 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:00.413415 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:00.414050 (+ 635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37832: BlockingRecv error: recv got EOF from 127.0.0.1:37832 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 08:58:00.417126 14505 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:58:00.416356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37834 (local address 127.23.167.253:36821)
0928 08:58:00.416389 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:00.416391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:00.416623 (+ 232us) server_negotiation.cc:407] Connection header received
0928 08:58:00.416715 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:00.416716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:00.416727 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:00.416752 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:00.417050 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37834: BlockingRecv error: recv got EOF from 127.0.0.1:37834 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 08:58:00.421319 14505 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:58:00.420274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37840 (local address 127.23.167.253:36821)
0928 08:58:00.420394 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:00.420396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:00.420666 (+ 270us) server_negotiation.cc:407] Connection header received
0928 08:58:00.420766 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:00.420767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:00.420779 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:00.420810 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:00.421250 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37840: BlockingRecv error: recv got EOF from 127.0.0.1:37840 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260928 08:58:00.427276 14505 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:58:00.426452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37846 (local address 127.23.167.253:36821)
0928 08:58:00.426491 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:00.426493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:00.426739 (+ 246us) server_negotiation.cc:407] Connection header received
0928 08:58:00.426837 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:00.426838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:00.426849 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:00.426878 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:00.427220 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37846: BlockingRecv error: recv got EOF from 127.0.0.1:37846 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 08:58:00.437697 14505 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:58:00.436599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37862 (local address 127.23.167.253:36821)
0928 08:58:00.436644 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:00.436646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:00.436965 (+ 319us) server_negotiation.cc:407] Connection header received
0928 08:58:00.437109 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:00.437110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:00.437124 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:00.437158 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:00.437635 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37862: BlockingRecv error: recv got EOF from 127.0.0.1:37862 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 08:58:00.456077 14505 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:58:00.455074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37872 (local address 127.23.167.253:36821)
0928 08:58:00.455130 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:00.455132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:00.455459 (+ 327us) server_negotiation.cc:407] Connection header received
0928 08:58:00.455564 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:00.455565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:00.455579 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:00.455608 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:00.456015 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37872: BlockingRecv error: recv got EOF from 127.0.0.1:37872 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 08:58:00.490227 14505 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:58:00.489496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37884 (local address 127.23.167.253:36821)
0928 08:58:00.489544 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:00.489546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:00.489639 (+ 93us) server_negotiation.cc:407] Connection header received
0928 08:58:00.489725 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:00.489726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:00.489739 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:00.489798 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:00.490164 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37884: BlockingRecv error: recv got EOF from 127.0.0.1:37884 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 08:58:00.555976 14505 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:58:00.555331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37894 (local address 127.23.167.253:36821)
0928 08:58:00.555406 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:00.555408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:00.555505 (+ 97us) server_negotiation.cc:407] Connection header received
0928 08:58:00.555552 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:00.555553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:00.555565 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:00.555616 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:00.555909 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37894: BlockingRecv error: recv got EOF from 127.0.0.1:37894 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 08:58:00.685829 14505 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:58:00.685146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37908 (local address 127.23.167.253:36821)
0928 08:58:00.685214 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:00.685216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:00.685294 (+ 78us) server_negotiation.cc:407] Connection header received
0928 08:58:00.685366 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:00.685367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:00.685380 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:00.685426 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:00.685762 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37908: BlockingRecv error: recv got EOF from 127.0.0.1:37908 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 08:58:00.943589 14505 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:58:00.942959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37914 (local address 127.23.167.253:36821)
0928 08:58:00.943023 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:00.943025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:00.943039 (+ 14us) server_negotiation.cc:407] Connection header received
0928 08:58:00.943121 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:00.943122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:00.943134 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:00.943185 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:00.943527 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37914: BlockingRecv error: recv got EOF from 127.0.0.1:37914 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 08:58:01.295517 14592 catalog_manager.cc:1259] Loaded cluster ID: 1da916ea05f74f42a0e0034fa3589a31
I20260928 08:58:01.295598 14592 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7f358adbe9764bea8a557f258f1ae61e: loading cluster ID for follower catalog manager: success
I20260928 08:58:01.296383 14592 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7f358adbe9764bea8a557f258f1ae61e: acquiring CA information for follower catalog manager: success
I20260928 08:58:01.296712 14592 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7f358adbe9764bea8a557f258f1ae61e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 08:58:01.364907 14639 catalog_manager.cc:1259] Loaded cluster ID: 1da916ea05f74f42a0e0034fa3589a31
I20260928 08:58:01.364988 14639 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ee83f0991eb240e6836da66fb4a37b49: loading cluster ID for follower catalog manager: success
I20260928 08:58:01.365823 14639 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ee83f0991eb240e6836da66fb4a37b49: acquiring CA information for follower catalog manager: success
I20260928 08:58:01.366099 14639 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ee83f0991eb240e6836da66fb4a37b49: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 08:58:01.459664 14708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:01.456885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37930 (local address 127.23.167.253:36821)
0928 08:58:01.457042 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:01.457044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:01.457080 (+ 36us) server_negotiation.cc:407] Connection header received
0928 08:58:01.457169 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:01.457170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:01.457223 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:01.457299 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:58:01.457462 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:01.457966 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:01.458804 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:01.459029 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:01.459089 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:01.459244 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:58:01.459377 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:58:01.459439 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:01.459550 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 08:58:01.459571 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":68,"threads_started":1}
I20260928 08:58:01.463749 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:01.461491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35726 (local address 127.23.167.252:46791)
0928 08:58:01.461589 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:01.461591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:01.461598 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:58:01.461635 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:01.461636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:01.461672 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:01.461739 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:58:01.461808 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:01.462274 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:01.463042 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:01.463247 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:01.463302 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:01.463409 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:58:01.463512 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:58:01.463566 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:01.463663 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 08:58:01.463680 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":51,"threads_started":1}
I20260928 08:58:01.468098 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:01.465780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35734 (local address 127.23.167.252:46791)
0928 08:58:01.465814 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:01.465816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:01.465823 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:58:01.465845 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:01.465846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:01.465854 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:01.465904 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:58:01.466010 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:01.466522 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:01.467390 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:01.467638 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:01.467695 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:01.467819 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:58:01.467853 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:58:01.467894 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:01.467991 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 08:58:01.468018 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 08:58:01.468209 14708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:01.465535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37934 (local address 127.23.167.253:36821)
0928 08:58:01.465580 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:01.465582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:01.465665 (+ 83us) server_negotiation.cc:407] Connection header received
0928 08:58:01.465731 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:01.465732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:01.465742 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:01.465791 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:58:01.465945 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:01.466660 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:01.467466 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:01.467768 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:01.467827 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:01.467999 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:58:01.468032 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:58:01.468072 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:01.468122 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 08:58:01.468143 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 08:58:01.468918 14721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:01.465502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49030 (local address 127.23.167.254:38589)
0928 08:58:01.465606 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:01.465609 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:01.465708 (+ 99us) server_negotiation.cc:407] Connection header received
0928 08:58:01.465826 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:01.465827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:01.465896 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:01.465953 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:58:01.466150 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:01.466894 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:01.467882 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:01.468188 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:01.468244 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:01.468538 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:58:01.468657 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:58:01.468710 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:01.468819 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 08:58:01.468839 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":52,"threads_started":1}
I20260928 08:58:01.470310 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 14357
I20260928 08:58:01.475580 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 14432
I20260928 08:58:01.479883 24223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN7pK8W/build/release/bin/kudu with pid 14507
2026-09-28T08:58:01Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2710 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5417 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (291485 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
2 FAILED TESTS